DOCUMENT RESUME 



ED 462 186 



PS 030 135 



AUTHOR 

TITLE 

INSTITUTION 

SPONS AGENCY 

PUB DATE 
NOTE 

AVAILABLE FROM 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Layzer, Jean I.; Goodson, Barbara D.; Bernstein, Lawrence; 
Price, Cristofer 

National Evaluation of Family Support Programs. Final Report 
Volume A: The Meta-Analysis. 

Abt Associates, Inc., Cambridge, MA. ; Yale Univ. , New Haven, 
CT. 

Administration for Children, Youth, and Families (DHHS ) , 
Washington, DC. 

2001-04-00 

118p . 

For full text: 

http : //www.acf . dhhs . gov/programs/core/pubs_reports/f amsup 
/fa m_sup_vol_a.pdf. 

Numerical/Quantitative Data (110) -- Reports - Evaluative 

(142) 

MF01/PC05 Plus Postage. 

At Risk Persons; *Children; *Effect Size; *Family Programs; 
♦Family (Sociological Unit); Meta Analysis; National 
Surveys; Outcomes of Treatment; Program Effectiveness; 
Program Evaluation 

♦Family Resource and Support Programs; *Family Support 



ABSTRACT 



This volume is part of the final report of the National 
Evaluation of Family Support Programs and details findings from a 
meta- analysis of extant research on programs providing family support 
services. Chapter Al of this volume provides a rationale for using 
meta-analysis. Chapter A2 describes the steps of preparation for the 
meta- analysis . Chapter A3 describes the 260 programs or interventions 
represented in the meta-analysis, examines their representativeness by 
comparing them with 167 family support programs that were not evaluated, 
describes characteristics of the studies included in the analysis, and 
compares them with excluded studies. Chapter A4 describes the analytic 
approach to answering the central research questions regarding the impact of 
family support services on selected child and adult outcomes and the program 
or treatment characteristics related to impacts. Chapter A5 details the 
findings of the meta-analysis. The analysis revealed that programs providing 
family support services had small but statistically significant average 
short-term effects on child cognitive development and school performance, 
child social and emotional development, child health, child safety, parent 
attitudes and knowledge, parenting behavior, family functioning, parental 
mental health and health risk behaviors, and economic well-being. Associated 
with stronger child outcomes were programs that targeted special needs 
children. Associated with less strong child outcomes were programs that used 
home visiting as their primary method of working with parents. Programs with 
the largest parent effects focused on developing parents' skills as effective 
adults: self-confidence, self -empowerment , family management, and parenting. 
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Introduction 



The Omnibus Reconciliation Act of 1993 (P. L. 103-66) amended Title IV-B of the Social 
Security Act and created Subpart 2 — ’’Family Preservation and Support Services.” This 
legislative initiative provides funding to States, through a capped entitlement, “for the purpose 
of encouraging and enabling each State to develop and establish, or expand, and to operate a 
program of family preservation services and community-based family support services.” The 
legislation defines family preservation and family support services as follows: 

• Family preservation services — services for children and families designed to 
help families (including adoptive, foster and extended families) at risk or in crisis, 
including services to help children at risk of foster care placement remain with 
their families, or where appropriate return to their families; 

• Family support services — community-based services to promote the well-being of 
children and families, designed to increase the strength and stability of families 
through such services as respite care, drop-in centers, early development screening 
and parent training. 

Funds were authorized for a five-year period, beginning with $60 million in FY 1994 and 
increasing over the remaining four years. Subpart 2, Section 435 of OBRA 1993 required the 
Secretary of Health and Human Services to evaluate the effectiveness of family preservation 
and family support programs. 

Although the two program types supported by the legislation share the ultimate goals of 
children’s and families’ well-being, they differ considerably in types of services provided, 
populations served, program philosophy and mission. They represent different points on the 
continuum of services provided for in the legislation. Family support programs offer services 
to all families in a given community (a neighborhood, a school, an early childhood program). 
Their hope is that, by supporting and building on family strengths, they can help families avoid 
stresses and conflict. For those families that are at risk, such services may ward off the 
problems that result in family break-up. There are tens of thousands of such programs and 
services across the country. For the most part, they are small, grass-roots efforts, developed 
within communities. Family preservation services, by contract, seek to assist families that are 
already manifesting problems or who are in crisis. The intervention is more specific, the 
population more defined, the programs fewer and more likely to be state-initiated. 

Acknowledging these differences, and seeking to respond fully to the legislative mandate, the 
Department of Health and Human Services determined that the two types of programs 
required different evaluation approaches. In September 1994, the Administration on Children, 
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Youth and Families awarded a contract to Abt Associates Inc. to document, describe and 
assess the effectiveness of family support programs and services. 

This is the first of three volumes of the Final Report for the National Evaluation of Family 
Support Programs. The three volumes report on distinctly different aspects of the study. This 
volume reports findings from a meta-analysis of existing research on programs that provide 
family support services. The second volume reports the findings from a small set of research 
studies of mature, well-implemented family support programs. The final volume is a set of 
case studies of the effect on the systems of family support services in three states of the 
additional funds for family support provided under federal legislation. 
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Chapter A1 

Rationale for the Use of Meta-Analysis 



The purpose of the National Evaluation of Family Support is to understand the effects of 
family support on families and children. A central task of the evaluation is to synthesize and 
analyze the body of research on family support, to answer the following questions, 

• What are the effects of family support programs and services on families and children? 

• How effective are different types of family support programs and services? 

• How effective are family support programs and services for children and families with 
different characteristics, needs and circumstances? 

There exist two strategies for summarizing the research on family support: narrative synthesis 
and meta-analysis. Narrative synthesis was, until recently, the strategy most commonly used 
in the social sciences for summarizing a set of research studies. Such reviews undertake to 
summarize the type and statistical significance of impacts reported in a group of studies of 
related programs. Conclusions are drawn about the distribution of effects, for example, 

“Eight of the 19 studies showed significant effects on children.” Sometimes statements are 
made about differential impacts related to characteristics of the programs or the types of 
participants. (For example, “home visiting programs that use professional home visitors more 
often report significant effects on maternal behavior”). 

Meta-analysis, on the other hand, is a quantitative synthesis technique rarely used in the social 
sciences before the 1970s. 1 In a meta-analysis, outcome results from individual studies are 
analyzed statistically to estimate an overall effect across studies. Meta-analysis allows us to 
go beyond simply counting up the numbers of studies with significant effects. Once all 
analysis results are transformed into a single common metric, that of the “effect size” 
(expressed for this study as a portion of a standard deviation unit), it is possible to combine 
and average data across outcomes measured on varying scales. We can also test for 
moderators of variation in effect sizes, such as programmatic or research design 
characteristics that may be related to differences in effect sizes. (For example, randomized 
experiments may be found to have larger effect sizes.) 

In the first year of the evaluation, we reported on a narrative synthesis of more than 75 family 
support programs (Barnes, Goodson & Layzer, 1995). While this report was widely 
circulated, it did not, and could not, in our opinion, constitute the “state-of-the-art knowledge 



1 Cooper, H.M., & Hedges, L.V, (1994). Research synthesis as a scientific enterprise. In F. Cooper & Hedges, L. (Eds). The handbook of 
research synthesis. New York: Russell Sage Foundation: p. 6. 
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base” that the original Request for a Proposal had specified. In addition, we recognized that, 
as we continued to gather more published and unpublished studies over the remaining four 
years of the study, a narrative synthesis would probably be inadequate for the task of 
summarizing the existing research accurately. Therefore, the decision was made, with the 
concurrence of the ACYF project officer, to undertake a meta-analysis of research studies and 
evaluations of family support programs conducted within the last 25 years. In addition to 
providing a more accurate assessment of the collective results of this research, the strategy has 
the advantage of producing a publicly-available database of studies that could, if desired, be 
continuously updated, and that would provide an analytic tool to address a variety of research 
questions. 

In the next chapter we describe the steps that were taken in preparation for the meta-analysis. 
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Chapter A2 

Preparation for the Meta-Analysis 



Preparation for a meta-analysis occurs in three stages, all three of which involve decisions that 
have important implications for the analysis and interpretation of findings. The first of these is 
Formulation of the Problem, in which the scope of the issue to be studied is defined. The 
second stage is Searching the Literature. This includes not only determining which studies 
should be included in the meta-analysis but also where to look for studies. The third stage is 
Coding the Studies which involves developing and implementing a coding system to describe 
the interventions, the research designs of the impact studies, and the impacts themselves. In 
the discussion below, each of these stages is described, together with the issues faced at each 
step along the way. 

Formulating the Problem 

Defining the issue to be studied is the starting point for a meta-analysis. The researcher is 
expected to start out with a “well-built” question. Such a question specifies: (a) the main 
interventions under investigation; (b) the participants or subjects of interest; (c) the outcomes 
of interest; (d) the comparison interventions of interest; and (e) the types of studies to be 
included (see Counsell, 1997; Meade & Richardson, 1997; Richardson, et al., 1995; and 
Oxman, Sackett, & Guyatt, 1993). 

Defining Family Support 

The first step in this process was to develop a working definition of family support, in order to 
determine which programs and interventions would be included in the analysis. Family 
support posed a number of challenges at this point. First and foremost was the lack of 
consensus on what constitutes a family support program or family support services. In 
addition, it was clear that, under any definition, a wide array of programs could be labeled 
family support, including programs that differ in their approaches to working with families, in 
what participating families actually do in the program, and in the length and intensity of family 
participation. 

There are at least two distinct ways of formulating a definition of family support as a basis for 
deciding which interventions are of interest. The first, which focuses on the types of services 
provided and their goals, is embodied in Public Law 103-66. This legislation defines family 
support as “...community-based services to promote the well-being of children and families, 
designed to increase the strength and stability of families, ...to increase parents’ confidence 
and competence in their parenting abilities, to afford children a stable and supportive family 
environment, and otherwise enhance child development.” (GAO, 1996) 
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A second kind of definition, widely used by family support advocates, is more philosophical 
and focuses not only on the goals of services but more particularly on the ways in which 
programs work with families to provide these services. In this definition, family support is 
seen as: normative — addressing issues faced by all families with young children; preventive — 
not designed to address specific problems or replace more intensive, professional services 
required by families in need of mental health treatment, remedial education or job training, or 
substance abuse treatment; and a mutually respectful partnership between family and staff 
that does not rely on professional diagnosis and treatment of pathology to discern individual 
needs. Multiple sets of practice principles have been developed by practitioners and experts in 
the family support field to guide programs, and to distinguish family support from other ways 
of worljang with families. 

The definition of family support contained in the federal legislation would include under the 
rubric of family support a wide array of services and interventions. It would include 
“traditional” family support programs whose primary mission is enhancing parents’ capacity to 
support children’s development, and which provide a variety of life skills workshops, 
parenting classes and parent support groups, parent-child groups and family activities, 
information and referral to other services outside the program, and advocacy for parents. In 
addition, the definition would include a set of more recent programs that have a primary 
mission other than enhancing parent capacity but which have incorporated family support into 
their programs as an integral part of their services. These programs may have the primary 
mission of serving adults through job training or other self-sufficiency skill building, or a child- 
focused mission such as preventing school drop-out, or a community development mission 
that focuses on housing or economic development. At the same time, these programs are also 
concerned with building families’ capacity to support their children’s development, and the 
services and opportunities offered to achieve the family support goals may be identical to 
those found in programs where enhancing parental capacities is the primary goal. 

In practice, it is not feasible to apply a practice-based definitional screen to the program 
descriptions contained in most research studies and evaluations since the definitions frequently 
vary. Often, it is not possible to apply such a screen to descriptions provided by program 
staff. Obviously, the definitions described above and earlier in the report were never intended 
to be used in this way. Rather, they were intended to serve as guides for program developers 
and staff, as well as, possibly, tools to evaluate the implementation of family support 
programs. 

With these considerations in mind, we elected to use a two-stage definition of family support 
for the purposes of the meta-analysis. First, to select programs to be included in the study, we 
applied a definition of family support that encompassed all services intended to improve child 
outcomes by strengthening the capacity of parents to support their children’s development. 
Under this definition, virtually all two-generation programs were included, as well as 
programs that provided family support services as an adjunct to adult-centered services or 
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child-centered services. Operationally, this meant that the meta-analysis included research 
studies or evaluations of any program or intervention intended to improve child outcomes that 
provided services to enhance parenting capacity, either as its central or as a secondary 
mission. 

Secondly, we rated all the programs included in the study, using a measure of 
comprehensiveness developed in cooperation with Sharon Lynn Kagan and other family 
support experts. This measure was used in conjunction with written information about 
program design and implementation to rate each program on how closely their practices in 
working with families match important practice principles in the field (e.g. the extent to which 
the program targets specific populations rather than offering universal services, whether 
participation is voluntary rather than mandated). The measure is described in more detail as 
part of our description of the coding process. 

Given the breadth of the definition we have adopted for initial inclusion of programs, this 
meta-analysis is vulnerable to the criticism that we are aggregating results from too diverse a 
sampling of studies. The fact that the review included such a diverse set of programs, varying 
in types of services offered and the methods of delivering these services, raises the question of 
whether a summary of effects across all programs and intervention will, in fact, be meaningful. 
One of the criticisms of meta-analysis is that it “mixes apples and oranges.” However, as one 
writer has noted: 

To some degree any synthesis of information from multiple research studies 
involves the aggregation of studies that are dissimilar. The same is true 
at the level of repeated observations of the same object. Some degree of 
mixing apples and oranges must occur in the tidiest of studies. Even when 
studies are intended to be direct replications, exact replication probably 
cannot occur. 

Hall et al., 1996, p.19 



The same writer notes a little later: 

..if the phenomenon [of interest] is conceptually broad and therefore should be demonstrated 
over a wide variety of contexts, then studies that vary extensively in subjects, 
situations and procedures may be appropriate for inclusion. 

Hall et al., 1996, p.20 

We believe that family support meets this test i.e., it is “conceptually broad and should be 
demonstrated over a wide variety of contexts”, and that our approach is defensible. In 
addition, and as another way of addressing concerns about the heterogeneity of programs, we 
developed a coding system that allowed us to distinguish among types of programs (where 
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“type”was defined and redefined in different ways) and to conduct separate meta-analyses 
within more narrow categories. 

Participant Groups of Interest 

In defining the participants of interest for the meta-analysis, the only limitation we imposed is 
that the program or intervention should be concerned primarily with the development of 
children between birth and age 12 years. Once this criterion was met, the review included 
interventions with all other participant groups. This meant that universal or non-targeted 
programs were reviewed for the meta-analysis, as well as programs for at-risk families and 
children. “At-risk” children included children at environmental risk, children at biological risk 
(low birth weight, premature, etc), and children who were identified as having behavioral or 
emotional problems. 

Outcomes of Interest 

Family support programs typically have a comprehensive set of goals, both in terms of the 
intended targets for change — children, adults, families and, sometimes, communities as well— 
and in terms of the breadth of the goals for each . This means that the research examines a 
wide range of outcomes for each group. It is not unusual for a single study to measure an 
extensive set of outcomes. A further complication is that outcomes are measured at different 
time points in the life of the intervention (during the intervention, at the end of the 
intervention, at various follow-up points) and at different points in the lives of the children 
involved in the intervention. 

Despite the difficulty of reviewing and summarizing data for a wide variety of outcomes, the 
research question of interest for the meta-analysis asks broadly about the impacts of family 
support, and therefore we were interested in all the outcomes reported in the research. 

Comparisons of Interest 

Two types of comparisons are potentially of interest for the meta-analysis. The first type of 
comparison tests whether outcomes for families and children who received the intervention 
are different at the end of the intervention from outcomes for families and children who did 
not receive that intervention. This comparison allows us to estimate the effect or “value 
added” of the intervention, and is at the heart of the meta-analysis. 

The second type compares the outcomes of different intervention approaches. For instance, a 
study might compare the effects of a home visit intervention that uses biweekly versus weekly 
visits, or professional versus paraprofessional staff. Studies that compare different 
interventions answer a different question from the one discussed above. They ask whether, 
for a given population, one form of intervention has larger impacts than another form of 
intervention. This is an interesting question, but to address it as part of a meta-analysis 
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requires that a sufficiently large number of studies compare the same set of program 
variations. 

Our decision, therefore, was to include only the first type of comparison in our analyses. The 
main meta-analysis will exclude treatment/treatment contrasts. At the same time, as part of 
the coding, we described the treatment/treatment contrasts in the research literature. If there 
are sufficient numbers of contrasts that test the same question, e.g., the relative effectiveness 
of home visiting versus center-based programming, of para-professional versus professional 
parent education staff, of two years versus one year of intervention, it would be possible to 
summarize the data on specific contrasts in a secondary meta-analysis. 

Types of Studies to be Included 

The question we faced here was first, whether to limit the scope of the meta-analysis to 
randomized studies or to include other study designs and secondly, whether any types of 
studies should be excluded from the review. Since it is generally agreed that randomized 
studies provide the strongest evidence about the effect of an intervention, many meta-analyses 
in other fields confine their search to experimental studies. 

While we recognized the analytic benefits of limiting the review to studies that used 
experimental designs, we were concerned that, in the field of family support, many of the best- 
known programs and certainly many of those seen as exemplifying the principles of family 
support have been evaluated with one or another type of quasi-experimental design. Limiting 
the review to randomized studies could mean that we systematically excluded studies of 
several different kinds of family support programs. 

After discussion of the issue with a group of experts in meta-analysis, including Dr. Thomas 
Cook, Dr. William Shadish, Dr Larry Hedges and Dr. Robert Fisher, we elected to include in 
the review both randomized and non-randomized studies but to record methodological 
variables that would allow us to examine the relationship between methodology and effects 
and to determine which specific categories of design we would exclude from the final 
analyses. 

Searching the Literature 

This step involved identifying strategies for collecting studies, conducting the searches and 
selecting studies. 
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Finding Research Studies 



The task of identifying all relevant research studies on family support services was a daunting 
one, since so many different fields of research fall under the omnibus definition of family 
support and because the research literature exists in so many forms and forums. It was crucial 
that we conduct a search that accomplished an unbiased if not complete identification of 
relevant studies. As one researcher notes, the data collection methods used (i.e., how relevant 
studies are identified) are “of primary importance to the results obtained in a systematic 
review or meta-analysis. ..[The validity of the results of statistical analyses depends on the 
validity of the underlying data.” (Dickersin, Scherer, Lebebvre, 1994: p. 1286) 

Exhibit A2-1 lists four ways of searching for studies that have been recommended for 
achieving “high” recall of documents. The goal, as described by White (1994), is to consider 
“a// empirical studies on a subject—not only the published but the unpublished ones— so as to 
capture in the syntheses the full range of reported statistical effects.” (p. 42) 

All of the methods listed in Exhibit A2-1 were employed for this meta-analysis. We searched 
for both published research and unpublished manuscripts. Although some researchers believe 
that unpublished studies should be excluded because they have not been peer-reviewed, an 
unmeasured influence on publication status is not assessment of scientific rigor but the nature 
of the results themselves. If fewer studies with negative or null results are published than 
studies with larger, more positive results, reviews that exclude unpublished works are likely to 
overestimate the relationship between an intervention and its outcomes (Dickersin & Yi, 1993; 
Dickersin, et al., 1987; Simes, 1987; Egger & Smith, 1995). This publication bias appears to 
hold true for small, nonrandomized studies in particular ( Dickersin & Yi, 1993; Easterbrook, 
et al., 1991; Newcombe, 1987). Most researchers conducting systematic reviews believe that 
unpublished studies should be included, and, if necessary, the results can be analyzed with and 
without the unpublished data (Cook, et al., 1993). 

As Exhibit A2-1 suggests, our search began, as an exhaustive literature search should, with 
research studies in published journal literature; these studies are the “most utilized by research 
synthesists” and the “most heavily represented in reference databases” (Reed and Baxter, 
1994). The less well-represented research literature formats include books and book chapters, 
research and technical reports, and conference papers. Even more “fugitive” are the results of 
studies which are not yet in any formal written report but exist instead as research memos or 
notes. 

These were found to a limited extent in some printed and computer databases, including: 

• ERIC, which lists published works but also some unpublished papers and papers 
presented at meetings and conferences; 
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The National Technical Information Service (NTIS), a database of summaries of 
completed research sponsored by 600+ federal agencies; 



• Social and Behavioral Science Documents (SBSD), published by the American 
Psychological Association, which contains abstracts of technical papers, reports and 
bibliographies; 

• Social Sciences Literature Information System (SOLIS), listing monographs, reports 
and conference proceedings in the social sciences and humanities (since 1976); and 

• Federal Research in Progress (FEDRIP), which provides access to information about current 
and ongoing federally funded research. 

Unpublished reports were also unearthed through the methods listed in Exhibit A2-1 under 
“Consultation.” Another rich source of literature was dissertations, which were searched 
through databases such as Dissertations Abstracts International, Master Abstracts, and 
Research Abstracts. In conducting the searches, the only criteria we imposed were: (1) the 
research was conducted after 1965; and (2) the research was conducted in the United States, 

Canada or Great Britain. 

Selecting Research Studies and Programs for the Meta-Analysis 
Database 

Out of the more than 900 research reports that were collected and reviewed, 1 we selected for 
coding 665 studies, representing 260 programs. In selecting which of the 900 studies to code, 
we took the approach that every relevant research study, where relevance was defined as 
involving an intervention, outcomes, and participants of interest, would be coded regardless of 
its design. All of the research studies in the data base compare two groups of participants 
who receive different kinds or levels of family support services. This includes: (a) studies that 
compare one group of families who receive the treatment (i.e., a specific set of family support 
services) with another group of families who do not receive this treatment; and (b) studies that 
compare a group of families who receive one set of family support services with another 
group of families who receive a different set of family support services. 



1 In addition, we obtained over 60 research reviews that summarize multiple studies on early childhood 

education (25 reviews), family support and parenting education (9 reviews), parent involvement (3 reviews), 
two generation programs (2 reviews), home visiting programs (4 reviews), Head Start (12 reviews), and child 
mental health programs (6 reviews). 
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Exhibit A2-1 

Recommended Strategies for Searching the Literature 

Footnote Chasing 8 

References in journals from nonreview papers 
References from nonreview papers not published in journals 
References in review papers written by others 
References in books written by others 
Topical bibliographies compiled by others 

Consultation 

Informal conversations with colleagues 
Attending meetings and conferences 

Communication with people who typically share information with you 
Formal requests of scholars who are active in the field 
Comments from readers/reviewers of past work 
General requests to government agencies 
Reviewing electronic networks 

Searches in Subject Indexes 

Manual search of abstract databases (e.g., Child Development Abstracts and Bibliography) 
Computer search of abstract databases (e.g., ERIC, Psychological Abstracts) 

Manual search of proceedings from relevant research conferences 

Citation Searches* 1 

Manual search of citation index 
Computer search of citation index 

Browsing 

Browsing through library shelves 
Browsing through colleagues’ shelves 

a Defined by White ( 1 994) as “the adroit use of other authors’ references to the prior literature on a topic.” 

b Following an author’s citations, forward and backward in time; examples include the Social Sciences Citation Index. 

SOURCE: Adapted from White (1994), in turn adapted from Cooper (1985) and Wilson (1992) 



In addition, we coded information on 167 family support programs for which there was 
descriptive literature but on which no outcome studies had been conducted (or which only 
provided data on participant satisfaction at the end of the program). We coded information 
on these programs to ensure that our descriptions of family support programs encompassed a 
broad spectrum of programs. 

The result of these decisions about which studies to code was a comprehensive database of 
studies that would allow us to describe a wide variety of programs. A list of the programs in 
the database and the research studies associated with them is provided in Appendix A. 
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Coding the Studies 



This project has two purposes: to describe family support programs, in order to characterize 
the field as a whole; and to summarize the research on the effects of family support programs 
using meta-analytic techniques. The coding system developed for the meta-analysis 
accommodates both of these needs by means of a hierarchical coding system. Exhibit A2-2 
shows the five levels of the coding system and describes the information that is coded at each 
level. At the “lowest” level of the coding system is the individual finding (contrast), which is 
the datum on which the meta-analysis is conducted. Higher levels of the coding system 
capture the study characteristics and the characteristics of the intervention, which may be used 
to explain differences in effects or to characterize the field of family support programs and 
research. 

At the base of this coding system is a specific contrast : numerical data describing the level of 
performance of two conditions (groups) on a specific measure at a specific time point. For 
example, a contrast might be the posttest means for a specific treatment and control group in 
a particular study on a child measure administered when the children were 12 months of age. 
The hierarchical coding system means that each contrast is linked to a large amount of 
information on (1) the measure itself, (2) the two groups being compared, (3) the study and its 
methodological characteristics, and (4) the program model being evaluated. 

The actual coding scheme was developed by researchers familiar with the research domain of 
family support. In developing the coding categories, we also consulted with experts in the 
area of research synthesis methods, who advised us about which study characteristics should 
be coded. A copy of the coding form and instructions can be found in Appendix B. 



Exhibit A2-2 

Levels of the Coding System for the Meta-Analysis of Family Support 



I. ProgramIModel: goals, services, target groups, length, delivery mode(s), staff qualifications 

A. Study : source of report, study setting, overall design 

1 . Contrast Conditions: details of sampling and attrition, presence or absence of threats to 
internal validity, description of family support services received by treatment condition 

a. Dependent Measure : name of measure, type of measure (adult vs child, self- 
report/observation/physiological measure), reliability of measure, level of performance 
of two conditions at pretest (data used to compute pretest effect size) 

(1) Time Within Dependent Measure: sequence in administration of measure (1st, 
2nd, 3rd, etc), age of child at administration, time elapsed since beginning of 
intervention, level of performance of two conditions at this administration of the 
measure (data used to compute posttest effect size) 
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Computing Effect Sizes 

In the research studies that we coded, the numerical data on which a contrast is based are 
presented in many different ways-as means, proportions, t-statistics, probability levels, etc. 

In our coding system, the data were recorded in the form that they were presented and were 
then converted into an effect size using specialized software developed expressly for this 
purpose by Dr. William Shadish. This software takes data expressed in 40 different forms and 
computes a standardized effect size using the appropriate statistical method. Appendix C 
briefly describes the forty methods handled by the software. 

The Coders and the Coding Process 

There were two separate coding activities. The first was the coding of the research articles in 
terms of the intervention itself, the study methodology, and the findings. The second activity 
was to rate each program in terms of its adherence to the family support practice principles. 

The research articles were coded by a small group of senior staff. Each coder was trained 
individually or in a small group by the senior researcher who developed coding protocol. The 
training took approximately six hours, and, at the end of the training, coders were required to 
complete a practice protocol that was checked for accuracy against a master coding. 

After the training, all articles were double-coded. Two coders completed a coding protocol 
independently, and then the two protocols were compared. Differences were discussed and, if 
necessary, given to the trainer for adjudication. An agreed-upon protocol was then entered 
into an Access database, to be converted into a SAS database for analysis. The exception is 
the outcome data (i.e., the actual numerical means, etc), which were entered separately into 
the program that computed the effect sizes. 

Rating Adherence to Family Support Practice Principles 

Each program or intervention studied was rated in terms of its adherence to family support 
principles. This rating was based on all available written information about the program. The 
rating system itself was developed in collaboration with Dr. Sharon Lynn Kagan of Yale 
University. The rating scale is discussed in Chapter A3. The actual rating of the programs in 
the meta-analysis was done by a small number of senior staff who are knowledgeable about 
family support programs and principles. 
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Chapter A3 

Characteristics of Programs and Studies Included in 
the Meta-Analysis 



The database constructed for the study contains information on 427 programs, all of which 
met our initial criteria for inclusion in the study. Of these, 260 programs had at least one 
evaluation study associated with the program; the remaining 167 had not been evaluated. 

Only the 260 programs that have evaluations are included in the meta-analysis, but we felt that 
it was important to code and include in the database programs that had not been evaluated. 
The inclusion of unevaluated programs in the larger database allowed us to represent a very 
wide range of program types and to compare the characteristics of programs that were and 
were not evaluated. The existence of systematic differences between the two groups could 
constrain the extent to which the findings of the meta-analysis might be expected to hold true 
for all types of family support program,, including those not previously evaluated. 

In this chapter, we first describe the 260 programs or interventions represented in the meta- 
analysis. Secondly, we address the question of the representativeness of these programs by 
comparing them with the 167 family support programs that were not evaluated. Finally, we 
describe the characteristics of the studies included in the analysis and compare them with 
studies that were excluded. 

Program Characteristics 

These descriptors represent characteristics of the programs or interventions themselves. In 
deciding on the aspects of programs about which we needed to code information, we chose 
those that met at least one of the following criteria: policy significance in the current debate 
about effective ways to work with families; relationship to program impacts demonstrated in 
previous research; or usefulness in describing differences across programs. 

The program variables selected can be grouped into ten categories: program goals and 
purpose; methods of delivering services to children and families; type and qualifications of 
program staff ; where services are delivered; whether the program is a research or 
demonstration program or a community-based program; whether the program targets a 
specific population and who is targeted; age of children toward whom the program or 
intervention is directed; the intended length of the program or intervention; the services 
offered; and the extent to which the program is rated as “family supportive". 
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Program Goals and Focus 

From the written program descriptions, we extracted the central stated purpose or goals of the 
program. As we might expect of family support programs, it was unusual for programs to 
have a single goal; most often their goals reflected their two-generation focus. Almost every 
program or intervention asserted the twin goals of improved parenting (98%) and enhanced 
child development (91%). Less than 10 percent of the programs had as a goal increasing 
family self-sufficiency, either by increasing parents’ educational or literacy skills or through 
job training or other employment assistance. Other kinds of assistance to parents, such as 
providing social support or developing self-help and leadership skills, were more likely to be 
articulated (22 percent and 12 percent of programs, respectively). Providing health care to 
children or to families was a goal for 17 percent of programs. Preventing child abuse was an 
explicit goal of only 14 percent of programs or interventions (Exhibit A3.1). 



Exhibit A3.1 

Program Goals 
(n=260 programs) 


Programs with Goal* 


Goal 


n 


% 


Improved parenting 


254 


98 


Child development 


237 


91 


Social support for parents 


57 


22 


Child/Family health care 


43 


17 


Child abuse prevention 


35 


14 


Parent self-help, empowerment - 


30 


12 


Parent literacy, employment 


22 


8 


Parent community/school involvement 


6 


2 


Child behavioral change 


6 


2 


* Written materials often cited multiple goals. 



Methods of Delivering Services 

Programs can deliver services in a variety of ways, including visits to the home, meetings or 
classes at the program or at other locations, or through written information. Almost half of 
the programs (49%) used home visits as the primary mode of service delivery, and another 12 
percent used home visits to deliver some services. More than half (59%) brought parents 
together in groups or for more formal classes at the program or another location and more 
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than a quarter (28%) provided joint activities for parents and children in a group setting. Only 
18 percent provided early childhood education services in a group setting (Exhibit A3. 2). 



Exhibit A3.2 

Mode of Service Delivery 
(n=260 programs) 




Primary Mode 


Secondary Mode 


Service Delivery Mode 


% of programs 


% of programs 


Home visits 


49 


12 


Parent meetings/classes/groups 


45 


14 


Parent-child classes/groups 


18 


10 


Group early education for children 


13 


5 



Staff 

The coding scheme for the meta-analysis allowed us to capture information on the staff who 
delivered services. For staff who worked with parents or families and for those who worked 
directly with children, we first determined their professional status (i.e. whether they were 
professionals, with both formal education and training, non-professionals, with a college 
education but no formal training, or paraprofessionals, without a degree or training before 
they were hired.) If professional staff were used, we determined the type, (e.g., medical, 
social worker, certified teacher, child development specialist). 

The majority of programs (87%) used professional staff to deliver services to parents (or to 
families). Almost one-third used paraprofessional staff, sometimes in combination with 
professionals. Few programs (9%) used staff in the intermediate category, that is, staff with a 
college education but no specialized training (Exhibit A3. 3). Less than 30 percent of the 
programs employed any staff to work directly and only with children. The majority of these 
programs, 70 out of 76, used professionals, sometimes supported by paraprofessional staff. 

More than one-quarter (28%) of the programs used social workers or trained counselors. 
Medical personnel, who might be doctors, nurses, physician assistants or physical therapists, 
were the next most frequently used professional staff (22% of programs). Only 17 programs, 
seven percent of the sample, had trained child development specialists on staff (Exhibit A3.4). 
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Exhibit A3.3 




Programs’ Use of Trained Staff 
(n=260 programs) 




Professional Status of Staff 


% of Programs 


Professional 

(degree and formal training) 


87 


Non-professional 

(degree, no formal training) 


9 


Paraprofessional 

(without training or degree) 


30 



Exhibit A3.4 

Types of Professional Staff Used by Programs 
(n=260 programs) 


Staff Type 


n of programs 


% of programs 


Medical (doctor, nurse, physical therapist) 


57 


22 


Social worker, counselor 


72 


28 


Certified school teacher 


50 


19 


Medical mix 


39 


15 


Non-medical mix 


25 


10 


Child development specialist 


17 


7 


Unspecified 


38 


15 



Location of Program Services 

Each program or intervention was coded as being based (i.e., services were delivered) 
primarily in the home, in a school, university or other institution of higher education, hospital 
or clinic, community center, or public or private social service agency. More than half of the 
programs delivered services in a single location; the remainder combined delivery of services 
at the program location with visits to the home. 

In a majority of programs (62%), some or all of the services were provided in the family’s 
home. Hospitals or clinics and schools were the next most frequent locations for services 
(29% and 25% respectively). Small numbers of programs provided services in other 
community locations (Exhibit A3. 5). 



0 
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Exhibit 3.5 

Primary Location of Program Services 
(n=260 programs) 


Location 


n of programs 


% of programs 


Family’s home 


162 


62 


Hospital or clinic 


75 


29 


School 


65 


25 


Community center 


42 


16 


University or college 


22 


9 


Public agency 


7 


3 


Private agency 


7 


3 


Other 


16 


6 



Type of Program 

We were interested in coding whether a program was identified in the study description as a 
research or demonstration program. Although they were not always explicitly categorized as 
such, we used the information provided in the study to determine whether the program was 
set up with a concurrent evaluation to test a strategy or approach and operated for a defined 
period. Many of these were single-site programs; others, like the Child and Family Resource 
Program were replicated concurrently in several sites, or like the Nurse Home Visiting 
Program had two successive single-site replications. 1 Our interest in this distinction stemmed 
from evidence from earlier research, in the field of early childhood education for example, that 
suggests that the effects of some research-driven interventions are stronger than the effects of 
subsequent programs generated at the local level that attempt to replicate all or some of the 
features of the original intervention. 

More than 70 percent of the programs included in the meta-analysis were classified as 
research or demonstration programs. To the best of our ability to determine, they had been 
funded and set up to test or demonstrate a strategy or intervention, and an evaluation was an 
integral part of their operation from the outset. Most ended at approximately the same time 



1 We recognize that Dr. Olds’ program has moved beyond the research and demonstration phase into multi-site 
and even state-wide replication. In the meta-analysis, however, the program is represented by studies of the 
original program and its two single-site replications. Since all the subsequent replications of the program must 
incorporate some level of research activity, in later versions of the meta-analysis, they would probably be 
categorized differently. 
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as the evaluation. While all remaining programs had been studied at least once, they were not 
set up for research purposes and had no defined period of operation. The majority of 
programs (73 %) operated in a single site. (This included most of the research and 
demonstration programs.) A quarter of the programs were in multiple states, and a small 
number of programs (2 %) were statewide multi-site programs. 



Exhibit A3.6 



Types of Programs in the Meta- Analysis 
(n=260 programs) 




Targeting of Specific Populations 

While many of the original family support programs were neighborhood entities that served 
any family in the neighborhood, their underlying principles and practices have been applied in 
programs that provide family support services to specific populations. We categorized 
programs as targeted if there was evidence that the program was designed for a specific 
population and if potential participants were screened for entry into the program or for receipt 
of services. If programs were categorized as “targeted”, the participation criteria were coded 
as either “environmental risk” or “biological risk”, the assumed risk being to the child’s 
healthy development. 

Environmental risk factors include: family poverty; welfare dependency; risk of or instance of 
child abuse or neglect; teen parenthood; maternal depression or isolation; incarcerated parent; 
and recent immigrant status, among others. Biological risk factors include: low-birth weight; 
physical disability; developmental delay; and behavior problems, among others. 
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As Exhibit A3. 7 shows, only 31 programs or 12 percent of the sample did not explicitly target 
their services to a specific population; more than half (55%) targeted families who faced one 
or more environmental risks. Families with a child at biological risk were targeted by 27 
percent of the programs, and a few programs directed services to families with both 
environmental and biological risk factors ( an example might be low-income families with a 
low-birth weight baby). 



Exhibit A3.7 

Program Targeting 
(n=260 programs) 



Of those programs that targeted families with environmental risks, a majority (63%) directed 
services to low-income or welfare populations. Families at risk for abuse and neglect were 
targeted by 22 percent of programs; teen parents were targeted by close to a quarter of 
programs. Developmental delay and low birth weight were the most frequently targeted 
biological risk factors (20% and 16% of programs, respectively). 

Ages of Children Targeted 

Most of the programs in the sample targeted children within a specific age-range. Half 
targeted children from birth (or before birth) to three years of age. Another 16 percent 
targeted children from birth to five years of age. Only 13 percent of the programs did not 
target a specific age group but provided services to children of all ages (Exhibit A3. 8). 



Environmental Risk 




No Targeting 



Mix of Environmental and Biological Risks j 
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Exhibit A3.8 

Ages of Children Targeted 
(n=260 programs) 



0-3 Yrs 




intended Length of Program 

Programs that had specified service periods provided services for an average of fifteen 
months. More than half of the programs or interventions (59%) were designed to provide 
services for less than a year. Of these, most were designed to provide services for six months 
or less. A very small proportion (6%) of programs did not specify a treatment period but 
were prepared to provide services to families for as long as the family wanted them or until 
the family reached a specified goal (Exhibit A3. 9). 
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Exhibit A3.9 

Intended Length of Program or Treatment 
(n=260 programs) 




Types of Services Offered 

Almost every program (98%) offered some form of parenting education, provided in a variety 
of ways. Most often provided through home visits, information about parenting strategies, 
children’s health and development and age-appropriate activities was also offered through 
classes for parents, classes for parents and their children, parent groups and printed materials. 
About half the programs organized parent groups and other activities to provide social 
support and reduce isolation. Almost half of the programs (44%) provided case management 
or counseling services or referral to social or health services. Just over one-quarter (29%) 
offered health care services for parents and children. Only 10 percent of programs offered 
adult education classes or other kinds of training aimed at improving parents’ economic 
status (Exhibit A3. 10). 
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Exhibit A3.10 




Types of Services Offered by Programs 
(n=260 programs) 




Types of Service 


% of Programs 
Offering Service 


Parenting education 


98 


Social support 


51 


Case management/counseling/referral to services 


44 


Support services (transportation, respite care, play 
groups, toy lending) 


36 


Health care for parents and children 


28 


Center-based ECE 


19 


Leadership/advocacy training 


16 


Adult basic education/job skills 


10 



Amount of Key Services 

Programs provided, on average, a total of 60 hours of parenting education to families. 
However, the average amount masks wide variation among programs. As Exhibit A3.1 1 
shows, about a third of the programs provided less than 20 hours of parenting education, 
another third provided between 20 and 40 hours, and the final third provided more than 40 
hours. Since 40 hours is roughly equivalent to half the number of hours in a typical 
community college course, most programs provided a relatively weak level of this service. 

Approximately 15 percent of the programs provided case management services to families. In 
these programs, families received, on average, less than 10 hours of case management 
services, with little variation across programs (Exhibit A3. 12). 

About one-quarter of the programs (59 programs) provided early childhood education 
services. In the majority of these programs (63%), children received less than 400 hours of 
early childhood education, or the equivalent of a day a week for a year. Another 15 percent 
provided between 400 and 1000 hours of early childhood education, while the remainder 
provided more than 1000 hours (Exhibit A3. 13). 

Intensity of Services 

The length of the program and the amount of service provided give us some measure of the 
level of “treatment” or “dosage” that families received, that may be helpful in understanding 
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Exhibit A3. 11 

Amount of Parent Education 
(n=260 programs) 




Total Number of Hours 



Exhibit A3. 12 



Amount of Case Management 

(n=37 programs providing case management) 




Total Number of Hour* 
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Exhibit A3. 13 



Amount of Early Childhood Education 
(n=59 programs) 




Total Number of Hours 



findings about program or service outcomes. Another aspect of “dosage” is the intensity with 
which services are delivered, assuming, for example, that the experience of 400 hours of a 
service spread over three years may have a different effect from the same number of hours 
concentrated in a shorter time period. For this description, the intensity of services was 
computed as the number of hours of a service divided by the number of months over which 
the service was provided. 

In the case of parenting education, about half the programs (5 1%) provided less than five 
hours a month of the service. Just over one-third (36%) provided between 5 and 15 hours of 
services and just over 10 percent provided more than 15 hours of services a month. (Exhibit 
A3. 14). 

Because the absolute amount of case management services was low, the intensity of the 
service was also low -- two-thirds of the programs provided one hour or less of care 
management services to families each month (Exhibit A3. 15). 

For early education services, almost 40 percent of programs provided less than half a day a 
month of services. At the other end of the spectrum were approximately 20 percent of 
programs that provided the equivalent of half-time to full-time early childhood education 
(Exhibit A3. 16). 
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Exhibit 3.14 

Intensity of Parent Education 

(n=260 programs) 




Exhibit A3. 15 

Intensity of Case Management 
(n=37 programs) 




Total Number of Hours 
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Exhibit A3. 16 



Intensity of Early Childhood Education 
(n=59 studies) 




! I T I 

4 Hours or Lose 4+ - 20 21+ -40 41+ -80 

Total Number of Houra/Month 



Family Supportiveness of the Program 

All of the programs included in the study met the definition of family support services 
provided in the federal legislation. However, we recognized that they were not all equal in the 
extent to which they embodied the principles of family support. Each study in the sample was 
rated on its “family supportiveness.” On the basis of work done by the Family Resource 
Coalition to develop a definition of family support, we developed a rating system that scores 
an intervention or program on seven dimensions. These seven dimensions do not comprise all 
of the features that, in the FRC definition, characterize family support programs. The seven 
were selected because they were features that we believed could be coded from the kind of 
program descriptions typically provided in research reports. The dimensions are as follows: 

• eligibility for the program; 

• presence of services to promote child development; 

• presence of services to promote adult development; 

• services focused on parent/child interaction; 

• services/activities that help build social support among parents; 
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* link to community resources (referral only or with active involvement with service 
providers in implementing service plan); and 

• community advocacy. 

Exhibit A3. 17 displays the criteria and the coding rules for rating the family supportiveness of 
a program. 



Exhibit 3.17 

Definitio n and Scoring Rules for Features for Family Supportiveness Score 

Definitions of Features Rating 



1 . Eligibility for the program is not based on identified 
problems or deficits, and the program is not 
designed to remediate specific problems. (For 
this definition, child age , ethnicity , geography are 
not counted as problems or risk factors). 

2. The program includes a component that focuses 
on the child’s development. 

3. The program includes a component that focuses 
on the development of the adult parent, including 
emotional, educational, economic development, 
life skills. 

4. The program includes a component that focuses 
on the development of the parent-child 
relationship. 

5. The program provides opportunities for peer 
support and builds informal networks of friends 
and neighbors. 

6. The program connects families to other resources 
in the community. 



No targeted eligibility: yes = 1 , no = 0 

Has child component: yes = 1, no = 0 
Has adult component: yes = 1 , no = 0 

Has parent-child component: yes = 1, no = 0 
Provides opportunities: yes = 1 , no = 0 
Connects families: 

2 = referrals + interagency cooperation 
1 = referrals only 

0 = no connection with community resources 



7. The program advocates for Community advocacy: yes = 1, no = 0 

improved/increased/more accessible services for 
families in the community. 



The total score for the measure was obtained by summing the scores of the seven individual 
items. Across the sample of 260 programs, the average score for family supportiveness was 
3.8 out a possible score of 8.0. The standard deviation for the score was 1 .48, indicating that 
96 percent of the programs had scores between 2 and 6. Half of the programs had total 
scores of 4.0 or greater. Almost all programs included both a child development component 
and a component that focused on the development of the parent-child relationship. Only six 
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percent of programs were scored positively on the item that measured advocacy for services in 
the community. Exhibit A3. 18 shows the distribution of scores for the 260 programs. 



Exhibit A3. 18 

Family Supportiveness Rating of Program 
(n=260 programs) 




Family Supportiveness Score 



Comparing Evaluated With Unevaluated Programs 

As we noted earlier, information on program characteristics was coded for both the 260 
programs with one or more evaluations associated with them and for 1 27 programs that met 
our criteria for inclusion, but had not been evaluated. We compared evaluated and 
unevaluated programs on the program characteristics described above: goals and services, 
targeting, service delivery methods, types of staff, length of program. Overall, the two groups 
of programs were significantly different on this set of characteristics (as shown in a 
discriminant function analysis). Specifically, although evaluated and unevaluated programs 
were similar with respect to their goals and types of services provided, compared with the 
unevaluated programs, evaluated programs: 

• were more likely to target their services to specific populations; 

• were more likely to use home visits as the primary mode of service delivery; 

• were less likely to use center-based early childhood education as a primary mode of 
service delivery; and 
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• were less likely to use paraprofessionals or non-professionals to provide parenting 
education. 

As Exhibit A3. 19 shows, the two groups of programs had similar goals. Improved parenting 
and enhanced child development were goals for almost all programs. One difference is that 
unevaluated programs were more likely to have increased community involvement as a goal 
for parents (19% vs. 2% for evaluated programs). The mix of program services offered by the 
two groups of programs was generally similar. Somewhat larger percentages of unevaluated 
programs provided parent social support and leadership training services, as well as center- 
based early childhood education. 

Only 12 percent of evaluated programs did not target services to a specific population 
compared with 53 percent of unevaluated programs (Exhibit A3.20). Evaluated programs 
were more likely to direct services to families with environmental risks or with children at 
biological risk, and to target specific age groups of children. 

Evaluated programs made more use of home visiting as a primary mode of service delivery 
(49% vs. 30%), and less use of center-based early childhood education (18% vs. 32%), as 
Exhibit A3 .21 shows. All programs had staff who provided parenting education, but Exhibit 
A3. 22 shows that evaluated programs were more likely to use professional staff to provide 
this service. Evaluated programs were somewhat more likely to provide services in the home 
(62% vs. 50%) and less likely to base services in a school (25% vs. 43%) or community 
agency (16% vs. 50%) (Exhibit A3 .23). 

The differences between the two groups of programs have implications for how we will 
interpret the findings reported in Chapter 5. While we may be able to generalize findings 
about the effects of specific service strategies, staffing patterns or targeting of services to 
any programs that provide these services, we need to be more guarded in the statements we 
make about family support programs as a group, recognizing that the programs included in the 
meta-analysis represent a substantial part, but not the universe of family support programs. 
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Exhibit A3. 19 

Primary Goals of Evaluated Versus Unevaluated Programs 




Exhibit A3. 20 

Targeting of Program or Intervention 




No Targeting 




[T1 Evaluated 
II Non-Eva tiatoO 




Environmental Risk 
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Exhibit A3.21 

Service Delivery: Primary Mode 




Home Visits Earty Childhood 

Classrooms 
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% of Programs 



Exhibit A3.23 

Location of Program Services 
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Chapter A-4 
Analytic Approach 



The meta-analysis addressed three central questions: 

What is the impact (i.e., average effect size) of family support services on selected child 
and adult outcomes? 

Are there program or treatment characteristics that are significantly related to the impacts 
(i.e., average effect sizes) of family support programs on child and adult outcomes? 

What aspects of family support programs or services account for variation in the impacts 
of family support programs on child and adult outcomes? 

This chapter sets forth our approach to answering these questions. The discussion begins by 
describing the analytic database used to conduct these analyses. 

Constructing the Analytic Database 

The analytic database for addressing the three research questions included only a subset of the 
studies on the full database. The final analytic database was constructed in three steps, 
moving from the full set of 665 studies and the 260 programs associated with 12,486 effect 
sizes, to two analytic databases: (1) a database for the end-of-treatment outcome data, 
which includes 5,681 effect sizes from 351 randomized or quasi-experimental studies of 191 
programs; and (2) a database for the follow-up outcome data, which includes 2,224 effect 
sizes from 158 randomized or quasi-experimental studies of 87 programs. 

In the first step, we excluded two kinds of studies from the full database. First, we eliminated 
studies that compared two different family support treatments. Second, we eliminated studies 
in which we did not have a true pretest to use in constructing an effect size. Eliminating these 
two types of studies reduced the database to 562 studies representing 255 programs. The 
data from these 562 research studies comprise a database of 1 1,1 12 effect sizes, in which each 
effect size measures, for a given study, the difference between two groups on a specific 
outcome at a particular point in time. The fact that most research studies of family support 
programs report data on multiple outcomes and at multiple time points accounts for the large 
number of effect sizes in the database. Effect sizes are grouped within studies, and, in this 
database, each study contributed, on average, 20 effect sizes, which represent different 
outcome measures, different time points, or both. 
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In constructing the database, we grouped the effect sizes within outcome domains as well as 
within studies. The effect sizes were grouped into nine broad outcome domains: four child 
domains— cognitive development/school performance, social-emotional development, 
health/physical development, and child safety; and five parent domains-parenting knowledge, 
parenting behavior, family functioning, adult mental health/health risks, and family economic 
self-sufficiency. Each of these nine outcome domains comprises a range of more fine-grained 
outcomes, listed in Exhibit A4. 1 . We then grouped effect sizes into three time points, 
reflecting when the outcome was measured: during the treatment period (interim); at the end 
of the treatment period; or at some point after the end of the treatment period (follow-up). 

Exhibit A4.2 shows the distribution of effect sizes in the database, by outcome domain and 
timing of the outcome measure. As the exhibit shows, the data base contained more parent 
outcomes than child outcomes. Of the child outcomes, the domain of social-emotional 
behavior was most frequently assessed; of the adult outcomes, parenting attitudes and 
knowledge were most often measured — nearly three times as often as actual parent behavior. 
Subsequent analyses to determine the effects of family support programs on children and 
families used the nine broad outcome domains, rather than the finer-grained sub-domains. 
This was both for practical purposes-to have sufficient data within a domain-and for policy 
purposes, since policy questions about outcomes for children and families tend to be 
enunciated in these same broad categories. 1 The exhibit also shows that the end-of-treatment 
outcomes were the most common in each of the outcome domains although, in the case of 
child cognitive outcomes, there were nearly as many follow-up effect sizes as there were end- 
of-treatment effect sizes. 

In the next step, we created two databases-one that comprised effect sizes associated with 
end-of-treatment outcomes (outcomes measured at the end of the intervention or service 
period) and a second comprising the effect sizes associated with follow-up outcomes 
measured after the end of the service period. 2 This resulted in databases of 6,860 end-of- 
treatment effect sizes from 488 studies and 2,502 follow-up effect sizes from 438 studies. 



1 Even the finer-grained outcome domains represent a much larger number of specific measures. The six sub- 
domains in Exhibit A4.I represent aggregations of 3,500 different instruments used to measure these outcomes. 

2 The interim outcomes were not included in these analyses, since we believed that the questions of most policy 
interest focused on outcomes at the end of the intervention or at an interval after the intervention ended. There 
are interesting research questions that could be addressed in future analyses that would examine when, during 
an effective intervention, the effects begin to manifest themselves. 
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Exhibit A4.1 

Outcome Domains and Components in the Meta-Analysis Database 



Child Outcome Domains 


Components 


Cognitive deveiopmenti 
s chooi performance 


Tests of cognitive development, language 
development; measures of developmental progress; 
aptitude tests 

School performance measures: achievement 
measures; school performance measures (grades, 
ratings of level of child’s performance, use of special 
ed services, promotion/retention/graduation 


Social-emotional development 


Measures of social skills with peers and adults, 
including responsiveness, positive/negative 
interactions; behavior problems, emotional stability, 
school behavior (truancy, absence, disciplinary 
actions), delinquency 


Physical healthlgrowth 


Birth circumstances, growth, illnesses, chronic health 
problems, receipt of preventive medical care 


Child safety 


Accidents, injuries, abuse, neglect, removal from 
home 


Parent/Family Outcome Domains 


Components 


Parenting knowiedgelattitude 


Parenting knowledge (developmental milestones, 
developmental expectations, etc), parenting attitudes 
(re discipline, communication, etc) 


Parenting behavior 


Parenting behavior, including home environment 


Family functioning 


Family functioning, family resources 


Health 


Mental health: locus-of-control, self-esteem, 
loneliness, sense of social support, depression, 
coping 

Health risk behaviors: pregnancy outcomes, smoking 
and substance abuse 


Economic self-sufficiency 


Educational progress/diplomas, job training, 
employment 
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The final decision about the database involved which types of study designs to include in the 
analyses. Of the 488 studies in the end-of-treatment database, 45 percent used random 
assignment designs, 27 percent used quasi-experimental designs, 3 and 28 percent used pre- 
post designs. 4 For the primary analyses, we elected to exclude the studies with pre-post 
designs, for methodological reasons (see Chapter 2). We then conducted tests, using the 
end-of-treatment data, to determine whether excluding pre-post studies introduced a bias into 
the sample of studies, that is, whether the programs evaluated with less rigorous methods are 
systematically different than the programs evaluated more rigorously. If this were true, it 
would mean that excluding the pre-post studies from the meta-analysis reduces the 
generalizability of the findings. 

To test whether the pre-post studies represented a different type of program from other 
studies, we compared, on a variety of characteristics, programs for which we had only pre- 
post studies with programs for which we had quasi-experimental and/or randomized studies. 
There was substantial overlap in the characteristics of programs with different types of 
evaluation studies, as Exhibit A4.3 shows. Nevertheless, the programs with only pre-post 
evaluations were, in fact, significantly different from the programs with quasi-experimental or 
the randomized evaluations: they were less likely to focus on children under 3 years of age or 
to use home visiting as the primary method of providing parent education, and they were 
shorter in duration, on average. 5 Excluding pre-post studies and the programs that were 
studied only with pre-post designs did not mean that the remaining sample failed to include 
some important part of the universe of programs serving families, only that the distribution of 
the resulting sample was different from the distribution of the full sample of programs 
identified in the meta-analysis. 

The final analytic databases included studies that employed either a randomized or a quasi- 
experimental design and that reported data from one or more of nine outcome domains. The 



3 Of the 120 contrasts with quasi-experimental study designs, about 40 percent employ an internal comparison 
group coming from the same pool of subjects as the treatment group (e.g. all started out in the same group), 
while about 60 percent are based on an external comparison group obtained from a patently different pool of 
subjects. 

4 Corresponding percentages for the follow-up database are 50 percent randomized designs, 33 percent quasi- 
experimental designs, and 18 percent pre-post design. 

5 Programs with quasi-experimental studies differed significantly from the programs that had been studied in 
experiments, although the differences were less substantial than in the contrast with the pre-post studies: 
Programs studied with quasi-experiments tended to have higher family supportiveness ratings and to more 
often be targeted to at-risk families. 
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Exhibit A4.3 

Characteristics of Programs by Type of Evaluation Design 3 



Type of Evaluation Design 



Program Characteristic 


Randomized 

(n=109 

programs) 


Quasi- 

Experiment 

(n=75 

programs) 


Pre-Post 

(n=60 

programs) 


Significant Group 
Differences 6,0 


Average family 
supportiveness rating 


3.55 


4.18 


3.69 


quasi > pre-post, 
randomized 


Focus on children < 3 
years 


67% 


57% 

o 


44% 


pre-post < randomized, 
quasi 


Includes an early 
childhood education 
component 


20% 


29% 


21% 


no s.d. 


Targeted to 
environmental risk 


57% 


72% 


56% 


quasi > pre-post, 
randomized 


Targeted to child 

developmental/biological 

risk 


39% 


28% 


33% 


no s.d. 


Targeted to teenage 
mothers 


12% 


22% 


19% 


no s.d. 


Home visiting as primary 
method of parent 
education 


59% 


49% 


27% 


pre-post < randomized, 
quasi 


Professional staff for 
parent education 


62% 


63% 


68% 


no s.d. 


Average # months of 
services 


13.76 

months 


15.36 

months 


6.67 

months 


pre-post < randomized, 
quasi 


All characteristics 
combined (multivariate) 








Pre-post significantly 
different than 
randomized, quasi 
combined (p < 01) 



a Each program was categorized in terms of the most rigorous study associated with that program; that is, if a 
program was studied with at least once using a randomized design, the program was placed in the Randomized 
group. 

b Between-group differences based on discriminant function analysis. 

c Design types separated by “ < “ are significantly different at p < .05 and lower; design types separated by a 
comma are not significantly different from each other. 



Abt Associates Inc. 



National Evaluation of Family Support Final Report A4-6 



45 



end-of-treatment analytic database comprised 5,681 effect sizes from 351 studies of 190 
programs (Exhibit A4.4). The follow-up analytic database comprised 2,224 effect sizes from 
158 studies of 87 programs (Exhibit A4.5). 

Level of Analysis 

The research questions for the meta-analysis focus on the study as the critical unit of analysis. 
That is, we are interested in understanding the effects of family support interventions, as they 
are implemented and studied at a particular time and place with a particular sample of 
individuals, and in determining whether variation in effects across studies can be predicted. 
Since the research questions are at the study level, it was appropriate to analyze the data at 
the study level. This meant that we needed to aggregate data from the level of the individual 
effect size to the study level. Our strategy for aggregating individual effect sizes is to average 
effect sizes within a particular study within an outcome domain. That is, instead of 
constructing a single effect size to represent all of the effects of a particular study, e.g., by 
averaging all of the effect sizes within a study, we instead constructed a separate aggregated 
effect size for each of the five outcome domains. In any outcome domain in which a study 
reported outcomes, we computed the average effect size for that study within that domain. 

As a result, an individual study is represented in the analytic database between one and five 
times, depending on the number of outcome domains for which that study reported data. In 
the database, a study could have an average effect size associated with child cognitive 
outcomes, child social-emotional outcomes, parenting outcomes, parent mental and physical 
health outcomes, and family economic self-sufficiency outcomes. 

With this strategy, theoretically each of the studies in the analytic database could contribute 
nine average effect sizes, one in each of the outcome domains. For the end-of-treatment 
database, which comprised 351 studies, this would result in a total of 3,159 average effect 
sizes in the database. In reality, the database included 1,004 average effect sizes across the 
351 studies, which means that each study had data in 2.8 outcome domains, on average. For 
the follow-up database, which included 158 studies, if each study had data in each of the nine 
outcome domains, the database would have included 1422 average effect sizes. The actual 
number of effect sizes was 361. 

Exhibit A4.6 shows the number of outcome domains represented in the studies in the end-of- 
treatment database. Over a quarter of the studies (28 percent) reported data in only one 
outcome domain. The majority of studies report on outcomes in three or fewer domains. 

Half of the studies (52 percent) reported both child and parent outcomes; just over a quarter 
of the studies (28 percent) reported only child outcomes, and the remaining 20 percent of the 
studies reported only parent outcomes. Virtually all of the remaining studies reported data on 
outcomes for both children and parents, with most studies reporting outcomes in only one or 
two domains for either group. 
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Exhibit A4.4 

Distribution of Effect Sizes in the Analytic Database: End-of-Treatment Outcomes 

(n=351 Studies; Randomized and Quasi-experimental Designs) 
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Exhibit A4.5 

Distribution of Effect Sizes in the Analytic Database: Follow-Up Outcomes 
(n=158 Studies; Randomized and Quasi-experimental Designs) 
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As a result of this approach to aggregating effect sizes, there were a different number of 
studies, each contributing an average effect size, in each of the outcome domains. In the end- 
of-treatment database, there were 187 average effect sizes in the child cognitive domain (i.e., 
effect sizes representing 187 of the 351 studies in the analytic database). The corresponding 
numbers of average effect sizes (or studies) in each of the other outcome domains were: 139 
average effect sizes for child social-emotional outcomes; 99 effect sizes for child health 
outcomes; 38 for child safety outcomes; 108 average effect sizes for parenting attitude 
outcomes; 166 effect sizes for parenting behavior; 32 effect sizes for family functioning; 144 
average effect sizes for parent mental health/health risks; and 91 average effect sizes for 
economic self-sufficiency outcomes (see Exhibit A4.4). 6 In the follow-up database, there 
were 101 average effect sizes in the child cognitive domain, 82 average effect sizes in the child 
social-emotional domain, 28 effect sizes for child health, 21 effect sizes for child safety, 22 
effect sizes for parent attitudes, 55 effect sizes for parenting behavior, 6 effect sizes for family 
functioning, 25 effect sizes for parent health, and 21 effect sizes for family economic self- 
sufficiency (Exhibit A4.5). 



Exhibit A4.6 

Number of Outcome Domains Reported on Studies in the Analytic Database 

(n=351 studies) 



Number of Outcome Domains 


Number of Studies (% of Sample) 


9 


0 


(0%) 


8 


3 


(1%) 


7 


31 


(9%) 


6 


10 


(3%) 


5 


27 


(8%) 


4 


30 


(9%) 


3 


44 


(13%) 


2 


106 


(31%) 


1 


97 


(28%) 



6 It should also be noted that in a number of bases, the same programmatic model was represented by multiple 
studies in the database. While these studies can be viewed as replications of programmatic models with 
varying levels of treatment intensity and methodological approach, for the analyses reported here, studies were 
treated as independent tests of family support interventions. Technically, stochastic dependencies amongst both 
outcomes within contrasts, and contrasts within studies, need to be accounted for in the analysis. This 
clustering effect, although of statistical interest, is often of little practical consequence when samples sizes are 
very large and estimates are thus often robust to these within-study dependencies (Will Shadish, personal 
communication). 
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Analytic Strategy 



Addressing the three primary research questions required three kinds of analyses. The first 
research question required description of the overall distribution of effect sizes within each of 
the outcome domains. The second research question called for analyses relating the average 
effect sizes within an outcome domain to individual program and treatment characteristics 
hypothesized to be predictors of the effect sizes. The third research question required 
developing and testing models using multiple predictors to explain differences in average 
effect sizes. All analyses were conducted twice, once with only the data from randomized 
studies and second with data from both the randomized and quasi-experimental studies 
combined. Further, parallel but separate analyses were conducted on the end-of-treatment 
outcomes and on the follow-up outcomes. (The only exceptions occurred in the case of the 
follow-up outcomes where, in some of the outcome domains, the sample sizes were too small 
to support multivariate modeling.) Each of these three analyses is discussed below. 

Distribution of Average Effect Sizes 

As described above, the analytic database comprised average effect sizes, with each effect size 
representing an aggregation of individual effect sizes .within a study in one of the nine 
outcome domains. To determine the distribution of average effect sizes within each of the 
nine outcome domains, we computed the overall average effect size within each outcome 
domain, by summing the average effect sizes across all studies and dividing by the number of 
studies with effect sizes in that domain. Second, we examined variation in average effect 
sizes among studies within an outcome domain to determine how much variation there was in 
these effect sizes. 

The extent of the variation in effect sizes determined whether we could continue on to the 
relational and modeling analyses. That is, analyses trying to explain variation in effect sizes 
only made sense if there was sufficient variation in the average effect size. To determine this, 
we conducted a statistical test (a test for heterogeneity of effect sizes) to tell us, in each 
outcome domain, whether there was sufficient additional variation among the true effect sizes. 
If the test indicated no significant heterogeneity of effect sizes for an outcome domain, we did 
not proceed with the relational and modeling analyses. 

Our strategy for estimating average effect sizes within the nine outcome domains was based 
on a random-effects regression model, rather than a fixed-effects model, because we wanted 
to generalize our findings to the population of studies of family support programs that is 
unlikely to have been fully represented by our sampled research (see discussion in Chapter 3 
about the representativeness of the sample of programs in the meta-analysis). Fuller 
discussion of the issues involved in using fixed versus random effects modeling is provided in 
Appendix D. 
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Single-Predictor Analyses 

For each outcome domain in which we determined that there was sufficient variation, we 
proceeded with analyses that tested the relationship between selected program and treatment 
characteristics (predictors) and average effect size. In these analyses, predictors were tested 
one at a time for each of the nine outcome domains. The results of these analyses told us 
which, if any, program or treatment characteristics account for a significant amount of 
variance in the average effect size of family support programs in different outcome domains. 

As Chapter 3 showed, the coding system we developed for recording information on each 
study in the meta-analysis provided us with data on many programmatic and treatment 
characteristics. From this large pool of variables we selected the set of variables to be tested 
as predictors. The variables selected as predictors were chosen based on findings from 
previous meta-analyses, advice from technical advisors, an understanding of which predictors 
would be most important to policy-makers, and on hypotheses we developed prior to the 
analyses. The predictors are listed in Exhibit A4.7. 

Treatment Predictors 

Treatment variables have been tested as predictors in virtually all meta-analyses of program 
effects. Whether the intensity of services received has any relationship to the size of the 
effects of a program is a question of interest to both policy makers and program designers and 
planners. We elected to test separately the effect of intensity of three kinds of services. Many 
family support programs provide a wide array of services to parents and/or children; there was 
little uniformity across programs in the combination of services provided. Therefore, we felt it 
was important to look separately at the effects of different kinds of service. The following 
treatment variables were selected for the analysis: 

• length of treatment (months elapsed from pre-test to post-test at end of treatment); 

• amount (hours) of services or treatment received by program parents and children 
during the treatment period— early childhood education services, parent education 
services, and case management services; and 

• intensity of services or treatment received (hours of service per month), for each type 
of service. 

Program Predictors 

The programmatic variables coded in the current meta-analysis include a large number of 
possible predictors. Variables in this group were selected a) so that we could address 
questions of specific policy interest (i.e., questions about the effectiveness of home visiting, the 
importance of staff qualifications, effects of different kinds of populations served), or b) 
because they had been found to be significant in previous meta-analyses of other types of 
programs, or c) because they represented some of the sources of variance in the sample of 
studies. The following program variables were selected for the analyses: 
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• program goals and focus; 

• services provided to families; 

• methods of delivering services; 

• whether program is targeted (vs universal) and types of parents or children 
targeted; 

• characteristics of population served by program; • 

• qualifications of staff who provide parent education; and 

• intended amount, length and intensity of services. 

Control Variables 

In these regression analyses, we also included a set of control variables, in order to have a 
more precise estimate of the relationship between a predictor and average effect size. The 
analyses then produced an estimate of the relationship of the predictor to average effect size, 
accounting for any variation that was associated with the control variable. These are termed 
control variables because, although the variables individually are not policy variables, we want 
to take their effects into account when we estimate the effects of the treatment and 
programmatic variables. Previous research has shown that these types of variables often 
account for a significant amount of the variance in effect sizes. Five variables representing 
sources of within-study variation were included as control variables. They are listed below 
and described in more detail in Appendix E: 

• accuracy of the method used to compute each effect size, 

• manipulability of the measure for each effect size (how easy it would be for the 
respondent to tailor his/her response to be closer to the “desirable” outcome, 

• number of effect sizes included in computing the average effect size for a study 
within each outcome domain, 

• sample size associated with each individual effect size across both treatment and 
comparison groups, and 

• where the data were reported (published or unpublished document). 

In addition, in analyses in which randomized and quasi-experimental studies were combined, a 
sixth cohort variable was added to account for the form of the study design. 

Multivariate Modeling Analyses 

In the third stage of the analyses, prediction models were developed and tested in order to 
determine a “best fit” model for each outcome domain. That is, different combinations of 
program and treatment characteristics were tested to identify the smallest set of predictors 
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that accounted for the most variance in the outcome domain. The modeling followed three 
steps: the selection of program and treatment characteristics to be tested in the modeling for 
each outcome domain; examination of the correlations among the potential predictors for each 
outcome domain; and development and testing of alternate prediction models for each 
outcome domain. 

Selecting the Variables to be Tested 

The selection of variables to be tested in the modeling analyses was based on the results of the 
single-predictor analyses of the relationships between each individual predictor variable and 
average effect size. Only the predictors that individually accounted for a statistically 
significant amount of variance in effect size were tested in the modeling analyses. Because we 
did not expect that the same predictors would be significant in the nine outcome domains, the 
single-predictor analyses were conducted separately for each domain and the significant 
predictors for each domain were carried into the modeling analyses. 

Examining the Correlations Among the Predictor Variables 

Once we identified the subset of variables to be tested as predictors in each outcome domain, 
we then examined the correlations among the program and treatment variables in each 
domain, to determine if any of the predictor variables were highly correlated with each other. 
The pattern of correlations among the program and treatment variables strongly influenced 
what models were tested, since we avoided entering highly-correlated variables in the same 
model. 

Developing the Prediction Models 

The modeling analyses had as their goal determining the “best fit” model for each of the 
outcome domains. The “best fit” model is defined as the model that: (a) accounts for the 
greatest amount of explained variance in effect size; (b) includes only predictors that 
accounted for a statistically significant amount of the variance, when the significant control 
variables and the other program and/or treatment predictors are also entered; and © includes 
the smallest number of predictors that accounted for the most variable in effect size. 

The process described below was repeated for data in each of the nine outcome domains. We 
began the modeling process by fitting models of effect sizes with two or more program 
variables (together with the control variables). The criterion for inclusion on the initial list 
was that the variable had to show a significant relationship with average effect size in the full 
sample. We fit a variety of models in each outcome domain, using different combinations of 
treatment and program variables. If a model was found to have two significant predictors, 
then additional models were constructed in which each of the remaining variables on the list 
was added to the previous model with two treatment/ program variables. Otherwise, a model 
with only a single significant predictor was retained as the preferred model. 
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The best models generally included predictors that were each statistically significant at the 
p< 05 level. The overall goodness of a model was judged by the proportion of variance 
explained by that model. If a significant amount of variance was unaccounted for after 
developing our best models, it suggests that if more, or more accurate, information were 
available on the measures, studies, treatments or programs, then better explanatory models 
could be found. 

In the chapter that follows, we present the results of these analyses. 
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Chapter A5 

Results of the Meta-Analysis 



In this chapter, we present the results of the meta-analysis in terms of the two major study 
questions: what are the effects of family support programs and services? and what 
characteristics of programs or the families they serve account for differences in effects? 



What are the Short-Term Effects of Family Support Programs and 
Services on Children? 



The broadest of the study’s research questions asked whether family support programs and 
services produce better outcomes for children and families. Improved outcomes for children 
are, in a sense, the ultimate goal of all family support programs. The population they serve 
consists of families with children; their services, whether focused on adults or on children or 
on the family unit, are intended to promote children’s wellbeing. As we saw in Chapter A-3, 
most of the programs included in the meta-analysis focused on families with children who face 
either environmental risks (most commonly poverty), biological risks, or a combination of 
both. Family support is widely viewed as a strategy for helping children overcome these risks 
and achieve their full potential. 

At the same time, rather than intervening directly with children, most of these family support 
programs worked primarily with parents, perceiving them as the agents of positive outcomes 
for children. While less than 20 percent of the programs provided early childhood education 
services directly to children, all but one or two provided parenting education, more than half 
offered social support services and more than 40 percent provided case management services, 
counseling and/or referral to services. This emphasis on parents reflects an important tenet of 
family support, articulated by Bernice Weissbourd in 1994, namely that “the capacity of 
parents to raise their children effectively is influenced by their own development” (Kagan & 
Weissbourd, 1994, p.32). In the same volume, Sharon Lynn Kagan sets forth guidelines for 
judging the quality of family support programs that include the following: 

• Programs recognize the importance of parental nurturing and seek to enhance parents’ 
capacity for growth and development: and 

• Programs understand that support can strengthen family coping capacities and strive to 
foster independence and empowerment. (Ibid, p.379) 

In recognition of the dual emphasis of family support programs, our analyses examined an 
array of outcomes for both children and parents. Child outcomes were grouped in four 
categories: cognitive development and school performance, social and emotional 
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development, health, and safety (injury, abuse, neglect). Parent outcomes were grouped in 
five categories: parent attitudes and knowledge, parenting behavior, family functioning; 
parental mental health and health risk behaviors; and economic well-being. Although no study 
reported outcomes in all nine domains, most reported outcomes in at least one child and one 
parent outcome domain. 

The results of the meta-analysis are presented below in terms of these nine outcome domains. 
Because a majority of studies reported outcomes at the end of the program only, this section 
focuses on the short-term effects of family support programs. A brief section at the end of the 
chapter discusses findings from the minority of studies that reported effects after follow-up 
periods of varying length. 

Summary of Findings 

Family support programs included in the meta-analysis, which represent a broad cross- 
section of programs that provide family support services, have small but statistically- 
significant average effects in all nine outcome domains. 

Small positive effects are found in the following areas: 

• Children’s cognitive development 

• Children’s social and emotional development 

• Parenting attitudes and knowledge, parenting behavior, and family functioning 

Statistically significant effects in favor of family support programs are found in the following 
areas, but the practical meaning of these effects is less certain: 

• Children’s physical heath and development 

• Children’s safety 

• Parents’ mental health or risk behaviors 

• Producing change in families’ economic self-sufficiency 

It is important to note that, in every outcome domain, a small group of programs 
accounted for the average effect; that is, in each outcome domain, more than half of the 
studies reported an effect size that was smaller than .20. Analysis of variation in 
program effects showed that: 

• Programs that focus on children with special needs have larger effects on 
children’s cognitive outcomes. 

• Programs that provide early childhood education directly to children have 
larger effects on children’s cognitive outcomes. 
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• Programs that provide parents with opportunities for peer support have 
larger effects on children’s cognitive outcomes; programs that use home 
visiting as a primary intervention have weaker effects on children’s cognitive 
outcomes. 

• Programs for families with children with developmental delays or behavioral 
problems that use professional staff to work with parents in group settings 
rather than through home visits have greater effects on children’s social- 
emotional development. 

• Programs for children at biological risk are least effective in producing 
positive change in children’s health and physical development. 

• Programs that target teenage parents with young children and combine case 
management with parent-child activities are more effective in protecting 
children from accidental injury, abuse or neglect. 

• Programs that use professional staff to help parents to be effective adults, 
and that provide opportunities for parents to meet in support groups, are 
more effective in producing positive outcomes for parents. 

• Programs that work with parents of children with special needs, and provide 
opportunities for peer support, have greater effects on parents’ attitudes 
towards and knowledge of childrearing and child development. 

Overall Effects 

Family support programs have small but statistically-significant effects in all outcome 
domains. These effects are evident when only randomized studies were included in the 
analysis and when both quasi-experimental and randomized studies are included (Exhibit 
A5.1). However, although each of the average effect sizes is statistically greater than zero (as 
shown by the significance of the T-tests in Exhibit A5.2), it is difficult to conclude that all of 
the effects should be interpreted as being educationally or psychologically meaningful. The 
convention in the social sciences is that effect sizes below .20 are not “educationally meaningful” (Cohen, 
1986). Effect sizes between .2 and .5 are considered small and potentially meaningful; effect 
sizes between .5 and .8 represent moderate effects; and only effect sizes larger than .8 are 
considered “large.” In three of the domains — child cognitive achievement, child social and 
emotional functioning, and parenting behavior — the effects are consistently meaningful (.2 or 
above in the two samples of randomized studies alone and combined with quasi-experimental 
studies) albeit small. It should be pointed out, however, that the “meaningfulness” of effect 
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sizes is not always clear, and must be considered in the context of evidence about (a) their 
potential importance for a given population and (b) the cost of obtaining the effect. 

Child Outcomes 

Cognitive Development and School Performance 

Family support programs, generally and in this sample, provide services to families with 
children ranging in age from newborns to children in high school. Not surprisingly, in the 
studies of these programs, a wide variety of measures are used to assess children’s cognitive 
outcomes including tests of cognitive and language development, aptitude tests, tests of 
developmental progress, school achievement and performance measures (grades, use of 
special education services, retention in grade, promotion, graduation). In our analysis, the 
effect sizes for all these different measures are grouped under the rubric of “cognitive 
outcomes.” However, since the majority of cognitive outcomes reported in the studies in the 
meta-analysis are for children under five years of age, most of the measures grouped under 
“cognitive outcomes” are assessments of children’s cognitive and language functioning rather 
than school-related performance measures. 

Family support programs have a small but positive effect on children’s cognitive outcomes. 
The effect size for the experimental and quasi-experimental studies combined is .29; when 
only experimental studies are included in the analysis, the effect is somewhat smaller — only 
.25. An effect size of .25 translates into a difference of four points on a typical standardized 
test of cognitive functioning with a mean of 100 and a standard deviation of 15; that is, 
children whose families received the family support services would score, on average, four 
points higher than children not in the program. 
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This effect is an average for children of all ages. The average effect of family support 
programs is greater when only outcomes for young children (under 5 years of age) are 
considered. For preschool children, the average effect is .39 across all studies and .35 for 
randomized studies only. This is similar to the size of the effect on cognitive development 
reported for the children in the Perry Preschool study at the end of the preschool year. 
(Weikart, Bond, & McNeil, 1978). 

Child Social and Emotional Outcomes 

Children’s social-emotional development is an important component of their readiness for 
school and also plays an ongoing role in their ability to function successfully throughout their 
school years. In addition, many programs focused on families with children whose 
development in this area might be compromised, e.g., families at risk for abuse or neglect or 
families with a child with special needs or with behavior problems. Measures of social- 
emotional development reported in the studies include: social skills; behavior problems; 
emotional stability; school behavior; and delinquency. 

Family support programs have a small average effect on social-emotional outcomes. The 
average effect size is .22 across all studies combined and .26 for the randomized studies 
analyzed separately. On a well-known rating of children’s social and emotional problems, the 
Achenbach Child Behavior Checklist, on the scale for externalizing (or acting-out problems), 
an effect size of .26 translates into a difference of three points on a 100- point scale, which 
could be large enough to make a meaningful difference in whether a child is judged to have a 
clinical level of problem behavior. 

Children ’s Health and Physical Development 

Children’s healthy physical development is frequently a goal of family support programs. 

Some programs have it as a primary goal, as in the case of programs for pregnant women 
aimed at reducing the incidence of low-birth-weight babies and other birth complications. 
Other programs are concerned with children’s health as part of a more global concern with 
promoting children’s well-being. In either case, improvement in children’s health and physical 
development is addressed primarily through (a) parent education about the importance of 
preventive health care for their children, good nutritional practices, home safety, etc.; or (b) 
referrals to medical care. A few programs offer medical care directly in the form of check- 
ups, developmental screening and other preventive care. A wide variety of health outcomes 
are reported in these studies, including growth indicators (height, weight), health (illnesses, 
hospitalization); and diet and nutrition. 

Family support programs have no meaningful effects on children’s physical health and 
development. The average effect size is .09 for the randomized studies and .12 when the 
randomized and quasi-experimental studies are combined. 
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Child Safety 

Although none of the programs in this sample are treatment programs for parents identified as 
having abused or neglected their children, a number of the programs are funded as child abuse 
and neglect prevention programs because they work with families defined as at-risk for abuse 
and neglect for a variety of reasons. These reasons include the presence of relatively specific 
risks such as evidence of inadequate or dysfunctional parenting practices, and more general 
risk factors such as poverty, teen parenthood, low level of parent education, isolation, and 
lack of social support. Despite the fact that many programs have as a goal reducing the 
likelihood that parents will abuse or neglect their children, most do not in fact measure the 
incidence of abuse or neglect. 1 Instead, the programs rely on measures of parenting behavior. 
Therefore, we have only a small number of studies with direct measures of child safety. These 
measures include reports of abuse or neglect, child removal for abuse or neglect, as well as 
reports of accidents, injuries and ingestions and hospitalizations due to these. 

On average, the programs have no consistently meaningful effects on children’s safety — .13 
for the randomized studies and .21 for the randomized and quasi-experimental studies 
combined. 

Parent Outcomes 

Parenting Attitudes and Knowledge 

The average effect of family support programs on parent knowledge and attitudes is small, 
about a fifth of a standard deviation. The average effect size is .23 when the randomized and 
quasi-experimental studies were combined, and smaller — . 1 8 — when only experimental studies 
were included in the analysis. Both of these effect sizes are on the edge of being considered 
educationally meaningful, according to convention. It is not clear whether a difference of this 
size represents a change that is large enough to have the effect on children’s well-being that it 
is ultimately intended to bring about. 

Parenting Behavior 

The average effect of family support programs on parenting behavior is also small — about a 
quarter of a standard deviation. The average effect size is .25 for the randomized studies 
alone and .26 for the randomized and quasi-experimental studies combined. It is difficult to 
assess whether an effect of this size would be meaningful on most measures of parenting. To 
try to understand its implications, we looked at observational data on parenting behavior taken 
from a large evaluation study of low-income families. The NCAST Teaching Scale is based 
on an observation of mothers teaching their children a simple task. In this study, the mean 



I The low incidence of maltreatment, even in high-risk populations, makes it difficult to detect the effect of an 
intervention, especially in small studies. 
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score for parenting behavior was 40.2, with a standard deviation of 5.6. 2 An effect of .25 
would translate into a difference of about one and a half points on the scale. 

Family Functioning/Family Resources 

A relatively small number of studies report measures of family functioning, limiting our power 
to detect relationships between effects and program characteristics. The average effect of 
family support programs on family functioning is more than a quarter of a standard deviation 
among the randomized studies. This effect size of .28 is considered educationally meaningful. 
However, the average effect size is much smaller for the randomized and quasi-experimental 
studies combined — .17. 

Parent Mental Health and Health Risk Behaviors 

The efforts made by many family support programs to provide or strengthen parents’ social 
support networks are intended to reduce feelings of loneliness and isolation and the depression 
that can accompany them and help parents to cope better with their lives. Other programs use 
a home visitor to accomplish some of the same ends. Improved coping skills and heightened 
self-esteem may result in reduced health risk behaviors such as smoking or substance abuse. 
Measures of all these types of outcomes are grouped together here. 

Family support programs have no meaningful effect on parents’ mental health. The average 
effect is .14 standard deviation for randomized and quasi-experimental studies combined, and 
even smaller for the randomized studies alone — .09 standard deviation. These effect sizes are 
so small as to call into question whether the programs are likely to make a meaningful 
difference on a measure of mental health outcomes. 

Family Economic Self-Sufficiency 

Only a few of the programs included in the study are job-training programs but, as family 
support programs increasingly targeted low-income families, the goal of helping families 
improve their life circumstances through continued education, job training or employment 
became more important and studies began to measure and report on educational and economic 
outcomes. Grouped together in this category are measures of educational progress and 
achievement, job training, employment and income. Family support programs have very little 
effect on parents’ economic well-being. The average effect size is 10 percent of a standard 
deviation. The average effect size is similarly small for randomized studies alone and when 
combined with quasi-experimental studies. 



2 Based on the sample from the evaluation of the Comprehensive Child Development Program (St. Pierre, 
Layzer, Goodson, & Bernstein, 1997). 
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Variation in Effects 



While the average effect size gives us a summary statistic about the effects of family support 
programs, it does not tell us about the amount of variation in effects across studies. The 
range in effect sizes in the nine domains is shown in Exhibit A5.3 (child outcomes) and Exhibit 
A5.4 (parent outcomes). In each of the domains, effects are unevenly distributed. Between 50 
and 60 percent of the studies report effect sizes of between -.19 and .19. The question we are 
interested in answering is whether there are systematic differences between the programs with 
positive effects and the programs that have essentially no meaningful effects. That is, are 
there patterns in the array of findings that point to features of programs that predict stronger 
effects? The analyses that were designed to answer these questions are discussed in the 
section that follows. 

It is important to note that these analyses of the effects of different program features were 
constrained by an important characteristic of the data. Despite the fact that there is substantial 
variation in effects across the studies in most of the domains, most of the variation in effects is 
not related to variation in program characteristics. On the contrary, most of the variation in 
effects is related to sampling error, or variation within studies. That is, while the expected 
value of the study's effect size is some common "true" effect, the estimates of that effect vary 
from study to study because of sampling error. In any study, the estimated treatment effect is 
a combination of the effectiveness of the intervention as well as any chance differences 
between the two groups. We tried to account for some of this within-study variation by 
including the “control variables” in each of our analyses. Once these sources of variation are 
accounted for, there is not much “true” variation left to be attributed to aspects of the 
intervention. This means that we cannot expect to identify program characteristics that 
explain a large portion of the variation in effect sizes. In fact, in one of the outcome domains- 
-family economic self-sufficiency — we could not conduct the relational analyses because there 
was not sufficient variation left to predict after accounting for the control variables (see the 
results of homogeneity of variance tests in Exhibit A5.2). 
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Which Program Characteristics are Related to Differential Effects of 
Family Support Programs and Services? 



Analyses were conducted to assess the strength of the relationship between individual 
programmatic, treatment and population factors and the magnitude of effects on outcomes. 
Exhibits A5.5 (randomized studies alone) and Exhibit A5.6 (for randomized and quasi- 
experimental studies combined) summarize the results of the analyses that individually tested 
each variable, i.e., each program, treatment or population characteristic, as a predictor of the 
size of a program’s effect in each outcome. Once the univariate analyses were completed, 
multivariate analyses were used to determine the combined effects of different combinations of 
program characteristics. 

In this discussion, it is important to remember two things about the database. First, program 
findings can be represented in more than one outcome domain. Although we describe findings 
in nine different outcome domains, we are really talking about nine overlapping, not 
independent, samples of programs. Therefore, findings from the separate analyses across the 
domains must not be interpreted as independent. Second, program characteristics are not 
themselves independent. That is, some characteristics occur together, at least in the set of 
programs in this meta-analysis, which makes it harder to disentangle the unique relationship of 
individual characteristics to effects. 

Because sample sizes within domains are sometimes relatively small, because we try to 
account for as many control variables as possible before estimating the effects of program 
predictors, and because we know that the total amount of variation we have to try to predict 
is small, the models we create are not complicated. Usually we can test the effects of only one 
or two program predictors simultaneously. 

Child Outcomes 

There are a number of significant relationships between program features and effects on 
children that are consistent across the two samples of studies (randomized studies alone and 
combined with quasi-experimental studies). Two program features stand out as being 
associated with stronger outcomes for children: 

• Targeting special needs children : Programs that target children with special 
biological or developmental needs have stronger effects on children’s cognitive and 
social -emotional outcomes. 

• Home visiting: Programs that use home visiting as a primary method of working with 
parents have smaller effects on child outcomes. 

The findings on types of program activities linked to effects are mixed, with different findings 
in different outcome domains. Early childhood education services are strongly linked to 
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cognitive outcomes for children but not to other outcomes. The amount and intensity of 
services are not related to any of the child outcomes. 

Cognitive Development and School Performance 

Four program characteristics are associated with the size of the effect on children’s cognitive 
outcomes: whether the program targets children at risk because of biological, developmental 
or behavioral problems; whether the program provides some direct early childhood education 
to children; whether the program relies on parent groups or home visits for delivering parent 
education; and whether the program provides opportunities for peer social support among the 
parents (Exhibit A5.5). 

As Exhibit A5.7 shows, the differences in average effect sizes associated with these four 
program characteristics are quite large. 

Programs that target children with physical or developmental disabilities, or children who are at 
risk because of low birth weight, have larger effects on cognitive development than other 
programs. Although most of the other programs also target at-risk children, they usually 
define risk in terms of environmental conditions associated with poverty and/or inadequate 
parenting rather than in terms of demonstrated biological, health or developmental problems. 
Programs that serve children at biological risk have higher average effect sizes than other 
programs, and the difference was large — 2.5 standard deviations difference (Exhibit A5.7). 

The finding that programs that provide early childhood education have, on average, larger 
effects on children’s cognitive development is not, by itself, a surprising finding. It simply 
replicates three decades of research on the effects of early childhood education. Among the 
randomized studies, the average effect size for programs with early childhood education 
services is substantially larger than the effect size for other programs. 
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Significant Relationships 3 between Programmatic Characteristics and Effects in Nine Outcome Domains: Randomized Studies 
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Exhibit A5.7 

Average Effects on Children’s Cognitive Development for Different Program Characteristics: 



Randomized Studies 


Program Characteristic 


Present 


Absent 


Effect Size of 
Difference 3 


Early childhood education 


.48 


.25 


2.1 s.d. 


Targeted to special needs children 


.54 


.26 


2.5 s.d. 


Peer support opportunities for parents 


.40 


.25 


0.9 s.d. 


Home visiting (vs parent groups) 


.26 


.49 


1.4 s.d. 



a The effect size is the number of standard deviations represented by the difference between the two means. 



The multivariate analyses show that the effects for early childhood education and for targeting 
biological risk are independent of each other. That is, early childhood education has an effect 
whether or not it is provided to children at biological risk. However, the difference associated 
with targeting is less dramatic among the programs with early childhood education (Exhibit 
A5.8). 



Exhibit A5.8 

Average Effects on Cognitive Development of Children with Biological Risks in 
Programs with and without Early Childhood Education: Randomized Studies 




Targeted to Children at 
Biological Risk 


Not Targeted 


Early childhood education 


.67 


.45 


No early childhood education 


.50 


.26 


Note: A difference of .05 represents an 


effect size of one standard deviation. 





Programs that rely on home visiting to deliver parenting education have smaller average 
effects on children’s cognitive development than programs that use parent groups. It is 
important to note that home visiting and early childhood education are strongly and negatively 
related in this sample of studies. Programs that provide early childhood education services are 
significantly less likely to use home visiting as the strategy for delivering parent education. 
Sixty percent of the programs that provide early childhood education services use home 
visiting, compared with 83 percent of the programs that do not provide early childhood 
education. In addition, programs that target children with biological or developmental risks 
tend to use home visiting less often than other programs. Therefore, we see a similar pattern 
of findings for home visiting as we did for early childhood education (Exhibit A5.9). 
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Exhibit A5.9 

Average Effects on Children’s Cognitive Development in Programs for Special Needs 
Children with Home Visiting vs. Parent Groups: Randomized Studies 



Primary method of delivering 
parent education 


Targeted to Children at 
Biological Risk 


Not Targeted 


Home visiting 


.36 


.09 


Parent groups 


.54 


.27 



Note: A difference of .1 1 represents an effect size of one standard deviation. 



The positive relationship between opportunities for parent peer support and effects on 
children’s cognitive development may also be mediated by the kind of child population 
targeted. Parent peer support groups are more common in programs that target special needs 
children, which may explain the positive relationship between peer support opportunities and 
effects on children’s cognitive development. 

No other programmatic features such as the program mission or the use of professional staff 
to provide parenting education are related to better cognitive outcomes. Although there have 
been concerns that paraprofessionals or non-professionals may not be as effective as 
professionals in delivering parenting education, the use of professional staff, trained in child 
development, to deliver parenting education does not predict better cognitive outcomes. 

Child Social and Emotional Outcomes 

Differences in effects on social and emotional outcomes are related to a large number of the 
program characteristics. Exhibit A5.10 shows effects are related to program goals, program 
services, how services are delivered, length of services, and the target group for the services. 

These predictors are highly inter-correlated, which means we cannot disentangle their effects 
to understand the contribution of any individual characteristic. However, the data indicate 
that programs fall into at least two clusters that are associated with average effects of different 
magnitudes. One cluster of programs targets children with developmental risks and/or 
behavioral problems, has as a goal the development of parent competencies, tends to use 
professional staff to work with parents, is less likely to use home visiting as a primary method 
of working with parents, and is less likely to work with primarily low-income families. These 
programs have larger effects on children’s social and emotional outcomes. A second cluster 
of programs uses home visits, employs at least some para-professional staff to work with 
parents, and tends to work with low-income families. This set of programs has smaller effects 
on children’s social and emotional outcomes. 
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Exhibit A5.10 



Average Effects on Children’s Social and Emotional Outcomes for Different Program 
Characteristics 





Randomized Studies 


Randomized & Quasi- 
Experimental Studies 


Program Characteristic 


Present 


Absent 


Effect Size 
of 

Difference 8 


Present 


Absent 


Effect Size 
of 

Difference 3 


Program goal: parent self- 
help-development 


.56 


.25 


5.2 s.d. 


.41 


.22 


1.9 s.d. 


Home visiting (vs. parent 
groups) 


.15 


.26 


2.4 s.d. 


.10 


.23 


1.4 s.d. 


Professional parent 
education staff (vs. 
Paraprofessional) 


.43 


.27 


4.0 s.d. 


.39 


.23 


1.6 s.d. 


Case management 
provided 


.08 


.27 


6.0 s.d. 


.08 


.23 


1.5 s.d. 


Targeted to children 
developmentally at-risk 


- 


- 


ns 


.39 


.22 


1.5 s.d. 


Serves majority low-income 
families 


- 


- 


ns 


.12 


.22 


1 .0 s.d. 


Program collaborates with 
other community agencies 


.08 


.26 


5.5 s.d. 


- 


- 


ns 


Opportunities for peer social 
support for parents 


.16 


.26 


1.5 s.d. 


- 


- 


ns 



a The effect size is the number of standard deviations represented by the difference between the two means. 



The multivariate analyses tested the variations in average effect size associated with 
combinations of these program characteristics. Exhibits A5.1 1 and A5.12 display the results 
of the analyses. Programs that have parent development as a primary goal and that use 
professional staff to work with parents (which are also programs that do not use home 
visiting) have an average effect size of .57, while programs with neither of these 
characteristics have an effect size of .23 (Exhibit A5.1 1). Programs that use case management 
as a service (which tend to be programs with more economically-related goals for families and 
which tend to work with low-income families) have, on average, lower effect sizes (Exhibit 
A5.12). 
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Exhibit A5.11 



Average Effects on Children’s Social and Emotional Outcomes in Programs with the 
Goal of Parent Self-Development with Professional vs. Paraprofessional Staff: 
Randomized and Quasi-Experimental Studies 


Parent Education Staff 
Qualifications 


Goal of Parent Self- 
Development 


Not a Goal 


Home visiting 


.57 


.41 


Parent groups 


.39 


.23 


Note: A difference of .08 represents an effect size of one standard deviation. 











Exhibit A5.12 



Average Effects on Children’s Social and Emotional Outcomes in Programs as a 
Function of Case Management, Staff Qualifications, and Program Goals: Randomized 
and Quasi-Experimental Studies 



Case Management Provided 


No Case Management 


Parent Education 
Staff Qualifications 


Goal of Parent 
Self-Development 


Not a 
Goal 


Goal of 
Parent Self- 
Development 


Not a 
Goal 


All professionals 


.43 


.25 


.55 


.41 


Some paraprofessionals 


.30 


.12 


.37 


.24 



Note: A difference of .07 represents an effect size of one standard deviation. 



Children ’s Health and Physical Development 

The earlier discussion showed that family support programs as a group have negligible effects 
on child health. The variation in effects across programs is, however, significantly related to 
one program feature, for both the randomized studies alone and when combined with the 
quasi-experimental studies: whether a program targets children with special needs (Exhibits 
A5.5 and A5.6). Programs that target children with special needs tend to have larger positive 
effects on children’s health and physical development than programs not targeted to special 
needs children (Exhibit A5.13). 
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Exhibit A5.13 






Average Effects on Children’s Health Outcomes for Programs Targeted to Children 
with Biological Risks 


Sample 


Targeted to Children 
with Biological Risks 


Not Targeted to Children 
with Biological Risks 


Randomized studies 


.57 


.41 


Randomized & quasi- 
experimental studies 


.39 


.23 


Note: For randomized studies, a difference of .06 represents an 
For the combined sample, a difference of . 13 represents 


i effect size of one standard deviation, 
an effect size of one standard deviation. 



Child Safety: Injury, Abuse, Neglect 

Only a relatively small number of studies measured child safety outcomes, which made it 
difficult to detect relationships between program characteristics and effect sizes. Because of 
the small sample of studies, the relationships between program characteristics and effect sizes 
had to be tested without including control variables in the models. 3 Larger effects on child 
safety outcomes are associated with programs that work with families of younger children 
(less than 3 years of age), with programs that provide case management services, with 
programs that provide parent-child activities, and with programs that work with teenage 
parents (Exhibit A5.14). Among these program features, serving families with younger 
children and providing parent-child activities are highly correlated, so that their individual 
relationships to effects cannot be disentangled. 

Among the randomized studies, the programs that target younger children and provide case 
management have, on average, an effect size of .86, while programs that target older children 
and do not provide case management have an average effect size of .13 (Exhibit A5.15). 

When the randomized and quasi-experimental studies are combined, the same pattern occurs. 
Programs that provide case management and either use parent-child activities or serve teenage 
parents have a very large average effect size — over 1.00— while programs with neither of 
these characteristics have only small effects (Exhibit A5.16). (These large average effect sizes 
must be treated with caution in light of the small number of programs in each of the samples 
and the large standard deviations in the effect sizes.) 



3 Based on the simple correlations between the control variables and effects sizes on child safety outcomes, none 
of the control variables was significantly related to effect size in this domain. 
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Exhibit A5.14 



Average Effects on Children’s Safety Outcomes for Different Program Characteristics 



Randomized Studies 


Randomized & Quasi- 
Experimental Studies 


Program Characteristic 


Present 


Absent 


Effect Size 
of 

Difference 3 


Present 


Absent 


Effect Size 
of 

Difference 3 


Target children 3 years and 
younger 


.56 


.13 


2.0 s.d. 


- 


- 


ns 


Case management 
provided 


.55 


.13 


1.9 s.d. 


.68 


.20 


1.4 s.d. 


Parent-child activities 
provided 


- 


- 


ns 


.86 


.21 


1.9 s.d. 


Targeted to teenage 
parents 


- 


- 


ns 


.61 


.20 


1.2 s.d. 



a The effect size is the number of standard deviations represented by the difference between the two means. 



Exhibit A5.15 



Average Effects on Children’s Safety Outcomes as a Function of Target Age of Child 
and Provision of Case Management Services: Randomized Studies 




Case Management 


No 


Age of Children 


Provided 


Case Management 


Targeted to children 3 years and 


.86 


.49 


younger 






Targeted to older children 


.50 


.13 



Note: In this sample, a difference of .19 represents an effect size of one standard deviation. 
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Exhibit A5.1 6 



Average Effects on Children’s Safety Outcomes as a Function of Provision of Case 
Management Services and a Teenage Parent Population: Randomized and Quasi- 
Experimental Studies 





Case Management 
Provided 


No 

Case Management 


Teenage parents 3 






Primary teenage parents 


1.04 


.58 


Not primarily teenage parents 


.66 


.20 


Parent-Child Activities 15 






Parent-child activities 


1.21 


.80 


No parent-child activities 


.62 


.21 



a A difference of .32 represents an effect size of one standard deviation, 
b A difference of .3 1 represents an effect size of one standard deviation. 



The multivariate analyses show that programs with all three features — case management, 
parent-child activities, and a teenage parent population, have the largest average effects — the 
average effect size for programs with this cluster of characteristics was very large — 1.40, 
compared with an average effect size of .20 for programs with none of these features (Exhibit 
A5.17). 



Exhibit A5.17 



Average Effects on Children’s Safety Outcomes Related to Provision of Case 
Management and Parent-Child Activities, and Serving Teenage Parents: Randomized 
and Quasi-Experimental Studies 



Randomized Studies 


Randomized & Quasi- 
Experimental Studies 


Age of Parents 


Parent-Child 

Activities 


No Parent- 
Child 
Activities 


Parent- 

Child 

Activities 


No Parent- 
Child Activities 


Serves primarily 
teenage parents 


1.40 


.90 


1.00 


.50 


Not primarily teenage 
parents 


1.11 


.61 


.71 


.21 



Note: A difference of .29 represents an effect size of one standard deviation. 
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Parent Outcomes 



For parent outcomes, the significant relationships among program features and effects are 
more scattered than they were for child outcomes (Exhibit A5.5 and A5.6). In general, the 
features most consistently related to the size of effects were the program goals and activities. 
The programs with the largest effects focus on developing parents’ skills as effective 
adults — their self-confidence, self-empowerment, family management and parenting. These 
programs also tend to provide opportunities for parents to meet in groups to provide peer 
support to each other. Programs that use professional staff to work with parents have 
stronger effects on parent outcomes than programs that rely more heavily on paraprofessional 
staff. 

Parent Attitudes and Knowledge 

One program characteristic is significantly related to effects on parent attitudes and 
knowledge, for both randomized studies alone and when combined with quasi-experimental 
studies: Programs that provide opportunities for peer support for parents have larger effects 
on parent attitudes and knowledge (Exhibits A5.5 and A5.6). The average effect size for 
programs that provide peer support opportunities is around .30, while the effect sizes for 
programs that do not provide these opportunities were around .20 (Exhibit A5. 19). Among 
the randomized studies, the other significant predictor is whether a program targets children 
with special needs. Those programs have an average effect size three times larger than the 
programs that do not target services to this group of children (Exhibit A5.I9). 

The multivariate analyses show that programs with both characteristics — targeting special 
needs children and providing peer support opportunities to parents — have an average effect 
size of .65, a large effect, while programs with neither characteristic have an average effect 
size (.17) that did not reach the level of being meaningful (Exhibit A5.20). When the 
randomized and quasi-experimental studies are combined, the second significant predictor is 
whether a program has a goal of parent self-development. These programs have a 
significantly larger average effect size — .35 versus .23 for programs that do not have this goal 
(Exhibit A5.18). In this sample of studies, programs that provide peer support opportunities 
also tend to have parent self-development as a goal; therefore we could not model their 
combined effects. 
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Exhibit A5.18 



Average Effects on Parent Attitudes and Knowledge for Different Program Characteristics 



Randomized Studies 


Randomized & Quasi- 
Experimental Studies 


Program Characteristic 


Present 


Absent 


Effect Size 
of 

Difference 3 


Present 


Absent 


Effect Size 
of 

Difference 3 


Peer support opportunities 
for parents 


.33 


.17 


2.3 s.d. 


.36 


.22 


1.9 s.d. 


Target children with 
biological risks 


.57 


.18 




- 


- 




Parent self-development as 
program goal 


- 


- 




.35 


.23 


1.4 s.d. 



a The effect size is the number of standard deviations represented by the difference between the two means. 



Exhibit A5.19 



Average Effects on Parent Attitudes and Knowledge as a Function of Targeting 
Special Needs Children and Providing Peer Support Opportunities for Parents: 
Randomized Studies 





Targeted to Special 
Needs Children 


Not Targeted to Special 
Needs Children 


Peer support opportunities provided 


.86 


.49 


Peer support opportunities not 


.50 


.13 


provided 






Note: A difference of .06 represents an effect size of one standard deviation. 



Parenting Behavior 

A cluster of program and treatment characteristics are related to the size of effects on 
parenting behavior (Exhibit A5.5 and A5.6). Three program features are related to effects for 
randomized studies alone and combined with quasi-experimental studies: whether the 
program has parent self-development as a goal, whether case management is provided, and 
whether the program is concerned with community change. As Exhibit A5.20 shows, their 
effects are as follows: 



• Programs for which parent development or self-help is a primary goal have an average 
effect size nearly twice as large as programs that do not. 
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• Programs in which community change is a focus have a significantly lower average 
effect size than programs that do not; whereas the programs with an emphasis on 
community change have essentially no effects, other programs have an average effect 
that is small but reaches a meaningful level. 

• Programs that provide case management services have smaller effects than the 
programs that do not provide case management. Programs that provide case 
management have essentially no effect on parenting behavior, while the average effect 
size for other programs is a quarter of a standard deviation — a meaningful effect. 

In the sample of randomized studies, an additional significant predictor of effects is whether or 
not the program has a goal of improving family economic self-sufficiency. In these studies, 
programs that focus on economic outcomes have significantly lower effects than other 
programs. 

For the randomized and quasi-experimental studies combined, one other program feature is 
related to the size of effects on parenting behavior. Programs that focus on families with 
children with behavior problems have much larger effects on parenting behavior than other 
programs. 

These predictors are themselves highly inter-correlated. The programs for which economic 
self-sufficiency is a primary goal are also the ones that tend to use case management and that 
tend to engage in community advocacy/change activities. These program features are all 
negatively related to effects on parenting. At the same time, these programs tend not to have 
parent self-development as a primary goal. They also tend not to be programs that focus on 
child behavior change. 

In the multivariate analyses, we examined the effects of combinations of program features. As 
shown in Exhibit A5.21, among the randomized studies, programs that provide case 
management (which also tend to focus on improving family economic self-sufficiency and to 
include activities focused on community change) have a very small average effect size (.11), 
while programs that focus on developing parent skills and capacities have an average effect 
size of .39 if no case management services are provided and .25 if these services are provided. 

Among the randomized and quasi-experimental studies combined, programs with the goal of 
developing parent skills and capacities, which also focus on families with children with 
behavior problems, have a very large average effect size — 1 .08 if no case management is 
provided and .97 if case management services are also provided (Exhibit A5.22). 
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Exhibit A5.20 



Average Effects on Parenting Behavior for Different Program Characteristics 



Randomized Studies 


Randomized & Quasi- 
Experimental Studies 


Program Characteristic 


Present 


Absent 


Effect Size 
of 

Difference 3 


Present 


Absent 


Effect Size 
of 

Difference 3 


Parent self-development as 
program goal 


.42 


.24 


.90 s.d. 


.49 


.26 


1.21 s.d. 


Case management services 


.08 


.25 


.87 s.d. 


.12 


.26 


.70 s.d. 


Community advocacy 
activities 


.04 


.25 


1.08 s.d. 


.10 


.26 


.80 s.d. 


Child behavior change as 
program goal 


- 


- 


ns 


.83 


.26 


2.76 s.d. 


Economic self-sufficiency 
as program goal 


.06 


.25 


.90 s.d. 


- 


- 


ns 



a The effect size is the number of standard deviations represented by the difference between the two means. 



Exhibit A5.21 



Average Effects on Parent Behavior as a Function of Providing Case Management 
Services and of Parent Self-Help as a Program Goal; Randomized Studies 



Program Services 


Targeted to Special 
Needs Children 


Not Targeted to Special 
Needs Children 


Case management services 


.25 


.11 


provided 






Case management services not 


.39 


.25 


provided 







Note: A difference of .19 represents an effect size of one standard deviation. 
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Exhibit A5.22 



Average Effects on Parenting Behavior as a Function of Case Management and 
Programs Goals of Parent Self-Development and Child Behavior Change: 
Randomized and Quasi-Experimental Studies 





Case Management Provided 


No Case Management 


Program Focus 


Goal of Parent 
Self-Development 


Not a 
Goal 


Goal of 
Parent Self- 
Development 


Not a 
Goal 


Focus on child behavior 


.97 


.75 


1.08 


.86 


change 

Not focused on child 


.37 


.15 


.48 


.26 



behavior change 

Note: A difference of . 1 8 represents an effect size of one standard deviation. 



Family Functioning/Family Resources 

A relatively small number of studies reported measures of family functioning, limiting our 
power to detect relationships between effects and program characteristics. One program 
feature is related to effects for both randomized studies alone and randomized and quasi- 
experimental studies combined. Programs that have child abuse prevention as a primary goal 
have larger effects on family functioning than other programs (Exhibit A5.5 and A5.6). The 
average effect size for programs that focus specifically on child abuse and neglect is 
large — .78 among the randomized studies and .52 when the quasi-experimental programs are 
included. The average effect size for other programs is .26 for randomized studies and .17 for 
randomized and quasi-experimental studies combined (Exhibit A5.23). Once again, programs 
that provide peer support opportunities have significantly larger effect sizes than other 
programs (Exhibit A5.23). 

Programs that focus on prevention of abuse and neglect also tend to be the programs that 
provide activities for peer support. Because of the inter-relationship among the predictors, we 
cannot say which of these characteristics is responsible for the stronger effects. 



O 
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Exhibit A5.23 



Average Effect Sizes on Family Functioning for Different Program Characteristics 



Randomized Studies 


Randomized & Quasi- 
Experimental Studies 








Effect Size 






Effect Size 








of 






of 


Program Characteristic 


Present 


Absent 


Difference 3 


Present 


Absent 


Difference 3 


Child abuse and neglect 


.78 


.26 


2.4 s.d. 


.52 


.17 


1.96 s.d. 


prevention as program goal 














Peer support opportunities 


- 


- 


ns 


.42 


.16 


1 .4 s.d. 


provided 















a The effect size is the number of standard deviations represented by the difference between the two means. 



Parent Mental Health and Health Risk Behaviors 

A number of program characteristics are related to the size of the effect on parent health 
outcomes, most in a negative direction (Exhibits A5.5 and A5.6). That is, a set of program 
features tended to be associated with very small program effects, including (a) a focus on 
improving family economic self-sufficiency; (b) provision of adult education services, and (c) 
provision of case management (Exhibit A5.24). In fact, these characteristics were 
correlated — a cluster of programs has all three characteristics, and this group of programs 
has essentially no effect on parent health and risk behaviors. The only program characteristic 
that is associated with larger and more meaningful effects on parents’ mental health, is use of 
professional staff. Programs that use professional staff to work with parents have an average 
effect size of .29, compared with an average effect size of . 14 among programs that use some 
or all paraprofessional staff. 

The multivariate analyses examined the simultaneous effects of three of the program features 
in the sample of randomized and quasi-experimental studies: program focus on economic self- 
sufficiency, use of professional staff, and targeting of children with special needs. The only 
combination of program features that produces meaningful average effect on parent mental 
health is use of professional staff to work with parents, for programs that do not focus on 
economic self-sufficiency and do not target children with special needs (Exhibit A5.28). 
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Exhibit A5.24 



Average Effects on Parent Mental and Physical Health Outcomes for Different Program 
Characteristics 





Randomized Studies 


Randomized & Quasi- 
Experimental Studies 


Program Characteristic 


Present 


Absent 


Effect Size 
of 

Difference 8 


Present 


Effect Size 
of 

Absent Difference 8 


Improved economic self- 
sufficiency as program goal 


-.07 


.09 


1.23 s.d. 


-.02 


.14 


1.0 s.d. 


Provides adult education 
services 


-.06 


.09 


1.15 s.d. 


.00 


.14 


.88 s.d. 


Provides parent-child 
activities 


- 


- 




-.06 


.14 


1.25 s.d. 


Professional parent 
education staff 


- 


- 




.29 


.14 


.88 s.d. 


Targeted to special needs 
children 


-.11 


.09 


1.43 s.d. 


- 


- 




Age of child at posttest 
(months) 


-.005 for each month of age 


-.003 for each 


month of age 



a The effect size is the number of standard deviations represented by the difference between the two means. 



Exhibit A5.25 



Average Effects on Parent Attitudes and Knowledge as a Function of Targeting Special Needs 
Children and Providing Peer Support Opportunities for Parents: Randomized Studies 



Focus on Economic Self- 
Sufficiency 


Not a Program Focus 




Professional 

Staff 


Some Para- 
professionals 


Professional 

Staff 


Some Para- 
professionals 


Peer support 
opportunities 
provided 


-.08 


-.25 


.11 


-.07 


Peer support 
opportunities not 
provided 


.13 


-.04 


.31 


.14 



Note: A difference of .16 represents an effect size of one standard deviation. 
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Family Economic Self-Sufficiency 

Most (80%) studies show no effect on family economic self-sufficiency, and only a small 
number of studies show a moderate or large effect. The amount of inter-study variation in 
effects was not significant. This means that not only was the overall effect on economic self- 
sufficiency small, there was too little variation in effects to try to predict them. 



What are the Long-Term Effects of Family Support Programs and 
Services on Children and Families? 

Follow-up data on outcomes after the end of services are reported in 158 of the 35 1 
randomized or quasi-experimental studies in the end-of-treatment database. Across these 
studies, the follow-up effects are measured as early as a few months after the end of services, 
up to as long as nine years after the end of services. In this database, the average length of 
follow-up is 3 1 months. Most of the follow-up data are obtained less than two years after the 
end of services, as shown below: 



Measurement of Follow-up Outcomes: 
Time Since End of Service 
(n=158 studies) 


6 months or less 


19% 


7-12 months 


22% 


13-24 months 


27% 


25 - 36 months 


6% 


37 - 60 months 


10% 


60+ months 


17% 



In the analyses that are described below, all of the follow-up data are combined. Length of 
follow-up period is tested as a predictor of size of effects and was found not to be significant. 

Overall Average Effects 

Exhibit A5.26 shows the average effects at follow-up in the nine outcome domains. There are 
only small samples of studies in most of the domains. For the family functioning outcomes, 
the sample size is too small to support analysis. For the randomized studies alone, the average 
effect sizes range from .04 for child health outcomes to .39 for economic self-sufficiency 
outcomes. Statistical tests indicate that the average effects are statistically significant in five 
of the domains (Exhibit A5.27). For the randomized and quasi-experimental studies 
combined, the average effect sizes are higher than for the randomized studies alone, ranging 
from zero for child health outcomes to .46 for economic self-sufficiency outcomes. The effect 
sizes are statistically significant in all of the outcome domains. 
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Program Characteristics Related to Differential Long-Term Effects of Family Support 
Programs and Services 

Exhibits A5.28 and A5.29 show the significant relationships between the individual 
programmatic predictors and the long-term effects in the nine outcome domains, for 
randomized studies alone and in combination with the quasi-experimental studies. For the 
randomized studies, there are very few significant relationships — so few, in fact, that we have 
to conclude that we do not know how to characterize the programs with larger long-term 
effects. When the quasi-experimental studies are included, there are more significant 
relationships, especially for the child outcomes. For this larger set of studies, case 
management is associated with smaller long-term effects on children. Another finding is that 
programs that provide peer support activities for parents tend to have smaller effects on 
children at follow-up. Parent/child activities, on the other hand, are positively related to long- 
term effects on children. In addition, programs that targeted children with biological risks 
tend to have larger long-term effects on children. 

Because there were so few statistically significant relationships between the programmatic 
predictors and follow-up effects, no multivariate modeling was done on the follow-up 
outcome data. 
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Average Adjusted Effect Sizes a,b in Nine Outcome Domains: Follow-up Outcomes 
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Weighted Mean Standardized Effect Sizes for Nine Outcome Domains: Follow-Up Outcomes 
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Exhibit A5.28 

Significant Relationships 3 between Programmatic Characteristics and Effects in Nine Outcome Domains: Randomized Studies 
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Significant Relationships 3 between Programmatic Characteristics and Effects in Nine Outcome Domains: Randomized Studies 
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Exhibit A5.29 

Significant Relationships 3 between Programmatic Characteristics and Effects in Nine Outcome Domains: Randomized and Quasi-Experimental Studies 
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Conclusions 



The goal of the evaluation in general, and of the meta-analysis in particular, was to investigate 
the effectiveness of family support services in improving outcomes for parents and, ultimately 
for children. The results of the meta-analysis offer some encouraging messages as well as 
some warnings. In addition, the findings offer some clues about how family support services 
might be strengthened. 

Family support services produce small but significant effects across a range of outcomes 
for parents and children. Family support programs and services are generally small-scale 
efforts with modest budgets. The levels of effects reported here seem, on the whole, 
consonant with the level of investment made in these programs. On the other hand, given our 
limited understanding of the practical meaning of these effects, one should use caution in 
making strong claims for family support as an intervention strategy likely to make a 
meaningful difference in families' lives. 

There is no single effective program model. Family support has been promoted as an 
effective strategy to address a host of social problems, from child abuse and neglect to school 
failure and delinquency. Although we have identified some strategies that have proven 
effective with specific populations, there is no single program approach, curriculum or service 
strategy that has demonstrated effectiveness across a range of populations. With one or two 
exceptions, the models that have shown larger effects have been tested in single-site research 
and demonstration initiatives and have not been widely replicated. 

The effects of family support are not evenly distributed across different program models 
and service strategies. There are hundreds of family support programs across the country, as 
well as thousands of schools, hospitals and other institutions that include family support 
services in the programs they offer. For most of them, the core service provided is some form 
of parenting education. It is sobering to realize that much of this effort may not produce even 
the modest effects that programs hope for. Almost two-thirds of the programs we studied had 
very small or no effects on parents' understanding of child development, attitudes about 
childrearing or behavior with their children. More than half of the programs had small or no 
effects on family functioning. 

Why are these services relatively ineffective? We can begin to understand some of the 
reasons, if we look at the strategies that produced stronger effects. Programs that use 
professional staff and deliver parent education and support through group meetings had 
stronger positive effects on parenting behavior and, in addition, on outcomes for children. 
Programs that focused services on specific types of families rather than on, for example, all 
low-income families in a neighborhood tended to be more effective. 
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However, family support service strategies have moved toward delivering such services 
through home visits, usually by paraprofessional staff. In addition, the family support 
philosophy emphasizes the desirability of non-targeted services. These strategies show the 
weakest effects on both parent and child outcomes. 

Family support services are effective in promoting children 's cognitive development and 
school readiness only if they provide services directly to children. The assumption that many 
parents lack the necessary skills to be effective teachers of their children has led to the 
widespread use of parenting education in family support programs. There is no evidence of its 
effectiveness in promoting children's cognitive development. Nor is it clear that adding parent 
education to direct services to children confers an additional benefit. Other major reviews of 
the relevant research found that adding parenting education to preschool programs did not 
increase their effectiveness. 

Family support services are effective with some important and vulnerable populations. 
Given the concerns that generated the federal legislation, it is encouraging that family support 
services that focused on teenage mothers with very young children, families that contain a 
child with special needs or families that have a child with behavior problems, all had strong 
positive effects on parents, on children or on both. It is important to recognize that, in 
addition to focusing services rather narrowly, these programs also tended not to use 
paraprofessionals to deliver services or home visiting as a service delivery strategy. Parent 
groups led by professional staff were important for parents of children with special needs and 
for parents of children with behavior problems. For teenage parents, organized parent-child 
activities were important. This was the one group of parents who appeared to benefit from 
having a case manager. 

The hearings on the legislation that provided additional funding for family support services 
produced testimony that promoted family support as an effective primary prevention strategy 
for child welfare. The findings from the meta-analysis make it clear that much more work 
needs to be done to identify effective ways to work with and support parents in their efforts to 
raise their children. The family support movement has made a significant difference in the way 
that hospitals, schools and social service agencies regard and treat families. It may need to 
re-examine some of its assumptions about the kinds of sendees that are truly useful to families 
for enhancing the development of their children. 
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